Given that we have a pretty tight container abstraction and live migration has come a long way, should we consider if live container (pod) migration isn't a better way to go? At the least, it probably deservers an "alternatives considered" entry.

We (and the user) would still need to deal with the node failure case, where live migration is impossible. So it becomes a matter of how many terminations the client sees, not whether they see them. Unless the reduction in number of terminations is dramatic and crucial (which I don't think it is), I think that the consistency of termination/failure semantics wins here (i.e. pods always terminate and replacements are created, rather than pods sometimes moving, and sometimes dieing and being replaced).

I would prefer we avoid live container (pod) migration, and I would imagine the rescheduler will continue to respect graceful termination.

+1 for avoiding live migration. Currently, Pods in k8s are stateless, we save data in PersistentVolume, so live migration is less meaningful.

+1 for avoiding live migration.

This proposal doesn't preclude migration, and this functionality would be required in order to implement migration.

However, migration would be a lot more work than what is described here.

Migration is covered by #3949.

Right. More generally, this proposal is an attempt to define an initial version and short-term roadmap, not the entire design space or long-term ideas. Once live migration of containers is available (the containerd docs seem to imply it will be soon? https://github.com/docker/containerd ) I would assume Kubernetes will take advantage of it.

@hurf Persisted state can still be "migrated" via persistent volume claim or flocker. Presumably the stateful applications have to be able to deal with restart after failure, so migrating the in-memory state shouldn't be strictly necessary.

Yes, that's what we do now. A problem is dealing with failure sometime may lower performance indicator of a service(doesn't mean not going to deal with the failure, but try to reduce failure). Especially in rescheduler case, failure may not caused by service itself but by an eviction(unless we give it a disruption budget of none). If we can have in-memory migration, the pod can get rescheduled without breaking its ongoing task. It frees more pods and looses the disruption budget restriction. Indeed it's not a necessary feature but an optimized option.

I'm not asking for live migration. It's a good thing but not urgent.

Could you be more specific here？I am a little confused. Does "garbage collecting" mean: Pod "Pa" appear in the PreferAvoidPods of Node "Na", but unfortunately scheduler schedule it to "Na" again, so after some period of time, Controller will try to delete it to see if there is a better Node for "Pa"?

Correct me I am wrong. IIUC, if the Pod "Pa" unfortunately goes to "Na" eventually, whether to re-evaluate the situation of "Pa" is not GC's work, it is completely another job. Once "Pa" is scheduled, it is scheduled.

The GC here means the removal of the controllerRef of the PreferAvoidPods, since this "anti-affinity" is by no means a permanent thing. Ideally, once the Pod we want to be running on the node gets correctly scheduled, the controllerRef of the evicted pod should be deleted, but it is hard to implement and kind useless. Just making controllerRef in the PreferAvoidPods a temporary thing is enough (by GC of the controllers and the api-server described below.).

Yeah, sorry for the confusing terminology. As @HaiyangDING inferred, I just meant the controller should remove itself from the list after some period of time.

I think we could use a admission controller to auto-generate a disruption budget. User could enable such an admission controller so that when they create a Service, a disruption budget will be auto-generated, or they can disable it and create disruption budget manually.

We can do just like LimitRange, users can create one, buf if they don't specifiy it, apply a default one.
Question: what's the behavior if this admission controller is disabled. All or none of the pods can be disrupted?

Is this really possible? If so, we should presumably fix it in the scheduler, not a rescheduler? If the scheduler schedules a pod to a node, and then discovers that it had out of date information about the node, and the pod can not in fact be scheduled there, it should automatically reschedule, possibly?

if a pod is scheduled to a kubelet and rejected in admission, isn't the pod immediately moved to a terminal state? i agree with @quinton-hoole here that this seems like something the scheduler should get right, but from what I can tell thus far in this PR is that the rescheduler does more selective killing of pods on a node, but does not actually take over primary scheduling responsibility so maybe its a no-op here and a solution is needed in both places?

In the model where scheduler has full information about kubelet resources, what you guys are saying is correct (and this is indeed how things are today). However, in the future it's possible that Kubelet will have more information than the scheduler, especially if the resource topology within a node becomes very complicated and it's not scalable for the scheduler to know all of the details. I would like to avoid moving to that world as long as possible, though.

+1 @davidopp 's comments.

@davidopp @HaiyangDING In my mind there are two distinct cases where a pod fails to schedule:

1. The scheduler picks a bad node, due to out of date or incomplete information. It's not clear to me that fixing this is the responsibility of a rescheduler.
2. The scheduler picks a non-optimal node, or no suitable node exists (without moving someother pods around). This seems like the purvue of the rescheduler.

Am I missing something?

@davidopp @HaiyangDING In my mind there are two distinct cases where a pod fails to schedule:

1. The scheduler picks a bad node, due to out of date or incomplete information. It's not clear to me that fixing this is the responsibility of a rescheduler. It seems like the scheduler should fix it's own mistakes here, e.g by trying again.
2. The node picked by the scheduler is non-optimal (or becomes non-optimal over time), or no suitable node exists (without moving some other pods around). This seems like the purview of the rescheduler.

Am I missing something?

I will try to answer with my knowledge, correct me if I am wrong.

The scheduler picks a bad node, due to out of date or incomplete information. It's not clear to me that fixing this is the responsibility of a rescheduler. It seems like the scheduler should fix it's own mistakes here, e.g by trying again.

Currently, if the pod is denied by the node proposed by the scheduler, the pod is simply marked 'failed', there is no trying again. I think in future:

1. It is the responsibility for the scheduler to try to scheduler the pod denied by kubelet again, or several times (<=N).
2. It is the responsibility for the rescheduler to handle the pod that has been denied by kubelet several times (>= N). However the mechanism needs further consideration. For instance, if the pod is denied by the same kubelet several times, we can add the avoid annotation on the node; but if the pod is denied by different, maybe we could just wait and see.

The node picked by the scheduler is non-optimal (or becomes non-optimal over time), or no suitable node exists (without moving some other pods around). This seems like the purview of the rescheduler.

Yes, it is. FWIW:

• The node picked by the scheduler is non-optimal (or becomes non-optimal over time), this is in the scope of the first version of rescheduler, but we need to figure out some policy to decide how non-optimal is bad enough (as well as the possibility to improve) to trigger the rescheduling behavior. Actually, by this we are asking the rescheduler to know (at least some of) the priority functions.
• suitable node exists (without moving some other pods around) , this is within the scope of rescheduler but is related to the preemption & priority, so they are not going to be implemented in the first step.
• Another use case of the rescheduler in the first step is to move some pod to under-utilized node, and both 'some pod' and 'under-utilized' need to be defined.

And finally, yeah, I don't think you miss anything :)

Yeah, I'd say this falls into the category of things that could go into either the rescheduler or every scheduler. As mentioned somewhere in the doc, we don't actually need a rescheduler component at all--we could just implement all of the rescheduler in every scheduler, creating a virtual/distribute rescheduler. But it's easier for people to write new schedulers (and the system is easier to understand, global policies are easier to configure, etc. etc.) if we have a single rescheduler component rather than putting the responsibility on every scheduler. With that in mind, the reasoning is basically what @HaiyangDING said -- while you could make schedulers responsible for noticing and stopping "rescheduling loops," you can instead make that the responsibility of the rescheduler, which would notice it happening (for any scheduler) and would add an indication to the node that equivalent pods should avoid that node for some period of time. But it is certainly the case that you could put this logic in the scheduler. (And we are not suggesting to address this at all in the first version of the rescheduler, especially since we don't have any scenarios today AFAIK that should cause rescheduling loops, other than stale information, which I don't consider a rescheduling loop because it will quickly stop.)

Let's move this to a design doc. It's underway already.

Nevermind. If we move priority out, isn't the rest resolved and underway, and this could be merged?

@bgrant0607 Issue/PR?

PodDisruptionBudget API is already merged, see
https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/policy/v1alpha1/types.go#L56

The controller for it is awaiting my review, #25921.

Last step is to implement /evict subresource (no PR for that yet).