The pods would have to have replication controllers for that to work.
@ddysher can say if decommisioning is currently possible, or what is planned, and how that would interact with stopping of pods.

Yes, having replication controllers is always the recommended way, but since Kubernetes allows for manual pod scheduling (with a pod descriptor instead of a replica controller descriptor) I think it would also be nice to support something like:

• Node is marked as scheduled for decommission
• Scheduler (?) is informed about this and reschedules pod(s) to other node(s)

We are not there yet. The best way is to use replication controllers for now.

After PR #3733 gets landed, I'll work on node lifecycle management and decommission. Node has a Terminated phase which is when k8s tries to re-schedule pods. This is for your first point.

For the second point, it'll be a little subtle to have all other components to understand node phase, especially the interactions between node controller, replication controller, and scheduler. I think the best way is to have node controller marks all pods as unscheduled, which triggers scheduler to re-schedule them. From the perspective of replication controller, the pod never fails, so we don't have to distinguish manual pod vs. controller-managed pods. There are probably more involved here, like restart policy, volumn, etc, will think it more.

I don't think we want to do the second point. Pods should not come back to life on a different node after they have been stopped on one node. We want the phase state machine to be a DAG not a general graph.

I personally think there's a reasonably-big usability gain from allowing pods to declare a restart policy that includes node failure and little downside. We already allow pods that are scheduled on a machine to restart as new containers if they crash, so the phase state machine already has cycles. And it seems strange to allow restarts only if containers on a node fail, but ban them when they'd start on a new node.

As a not-so-strawman example, setting "onNodeFailure: reschedule" seems like a reasonable user request.

Do we have cycles in current state machine? Restarting containers seems to be a cycle, by containers are not primitive in k8s. We do create Pod on failure from replication controller, but the newly created Pods are not the same entities as previous ones, they have different UIDs at least.

What the second point (re-schedule) really sounds like is migrating the Pods from failed node to new node. This is tricky, not to mention volumn, sessions. I have no objections to the DAG approach, but I agree with @alex-mohr that we need to do something here for usability. To do this, we'll at least need to distinguish manually-created Pod and replication-created Pod, maybe reverse label lookup? I don't know if that ever works.

We need to be able to drain machines. We should use an approach consistent with other resources. See #1535 re. graceful termination.

@alex-mohr We've discussed this before. Pods are replaced by replication controllers. They are not rescheduled. In fact, I'd like the replication controller to be able to replace pods ahead of killing the ones they are replacing in the case of a planned move. Let's not derail this issue with the "forever pod" discussion.

At the moment, the system doesn't have reasonable behavior in the cases of either planned or unplanned node outages. Let's fix that -- #1366 covers the unplanned case. This issue can cover the planned case.

@ddysher Why do you want reverse label lookup? The node controller needn't be aware of pod controllers -- replication controller, job controller, etc.

It would be great to have something in kubectl that allowed you to mark a node to be pulled out of rotation (and then showed up with kubectl get minion) and then drained after being marked for deletion or maintenance.

I ran into an issue like this the other day where I needed to rotate out some hosts for maintenance and had to manually remove pods after stopping the servers.

@bgrant0607 The reason I'm trying to do so is the possible overlapping functions between node controller and replication controller or job controller.

If all pods are started with replication/job controller, then node controller just needs to remove the pods. But cases like the issue where pods are started without any controller, node controller should be responsible to remove them and recreate them elsewhere. The restart part seems to be a duplicate function, ie. node controller and replication controller will alll try to create a new pod.

If node controller just removes the pod, then this seems to break our restart policy. A user would want a pod with restartalway to always restart even in case of node failure.

Did I interpret it correctly? I can't recall of any component that would claim 'ownership' of those pods.

@ddysher No, the node controller should never recreate pods elsewhere. That's not its job. Users that want that behavior need to run the pods under a pod controller, such as the replication controller. No, it doesn't invalidate restart policy -- separation of concerns. Individual pods are vulnerable to node failure -- that's reality and the model. See #3949 for more details.

The model makes node controller much easier, and that's definitely a good thing. But from a user's perspective (not from how we design/simplify the system), node failure without Pod restart is really confusing. We haven't stressed enough about pod controller, even our classic example creates naked pod, as you mentioned in #1603.

Here I'm not saying we should recreate pods, just bring up a potential issue if we don't do so :)

Yes. we should fix our broken examples.

I think you should remove the notion of a pod without a replication
controller. It would simplify the possible scenarios.
On Feb 4, 2015 4:47 AM, "Brian Grant" notifications@github.com wrote:

Yes. we should fix our broken examples.

—
Reply to this email directly or view it on GitHub
#3885 (comment)
.

@pires Been there, done that. Pod needs to be an available primitive. One reason is that we plan to support multiple types of controllers.

I think ability to mark the node as deactived/decommissioned using kubectl gives flexibility to the user/admin to do node upgrades(security patches, software upgrades), node evacuation/custom pod migration.
To support this use case, may be we can add a new condition on the node, say 'NodeDeactivate'. When 'NodeDeactivate' status is set(Full), irrespective of NodeReady/NodeReachable condition status, scheduler can ignore this node for new pod creation.
CLI can be:
Node Deactivation:
kubectl update nodes --patch={'apiversion': , 'status': {'conditions': [{'kind': 'Deactivate', 'status': 'Full'}]}}

Node Activation:
kubectl update nodes --patch={'apiversion': , 'status': {'conditions': [{'kind': 'Deactivate', 'status': 'None'}]}}

I'm planning to implement this feature, let me know if you see any issues with this approach.
@bgrant0607 @smarterclayton @ddysher @alex-mohr

@pravisankar Discussion on #1535, #2315, and #2726 is relevant.

Status must be completely reconstructable based on observations. In order to express that the desired state is "deactivated", there would need to be a field in NodeSpec that indicates this. There can additionally be a NodeCondition that reflects the status.

I imagine we'll eventually want several flavors of this:

• stop: gracefully terminate and remove the node -- we need to define what that means, but I'd like some reasonably consistent, useful definition of this for every object; we'll at least want to wait for pod pre-stop hooks to complete
• unschedulable: simply don't schedule new pods/volumes, but don't delete existing ones
• uninhabitable: evict existing pods/volumes
• shutdown: actually do something physical to the machine, like invoking shutdown

Note that there's some amount of nuance in some of the above. We will eventually want to differentiate between different types of workloads, such as daemons vs. user pods.

I eventually want a custom control subresource to set whatever such fields we add, as discussed in #2726, but patch would work initially.

For the next iteration, it would be nice to have a way to provide specific resources to drain. In the discussion about GPUs in #19049, I mentioned specific kernel driver ABIs. Draining a whole machine works, but if all we need is kicking out GPU users, maybe we could evict just the pods using the resource. Similar reasoning if you wanted to reformat attached SSDs, etc.

That could be achieved with taints and a rescheduler. At least taints will make probably it into 1.3. I think what we planned for 1.2 is complete.

#17190

ref/ #22217

When considering the maintenance mode use case, it'd be good to account for the ability to schedule the node drain. In this way, administrators may set a predefined maintenance window for specific nodes.

Are Jobs a good candidate to orchestrate a maintenance window once node drain is implemented?

Your analogy is reasonable, that scheduling node drains is similar to scheduling a workflow of run-to-completion Jobs. But I don't think the Job abstraction can be directly used for the former. More generally, I'm not sure how much sophistication for scheduling maintenance workflows we want to build into core Kubernetes, vs. suggest people build it on top. My initial thought is that we want to support simple server-side drains (something that marches through the cell at a specified rate and respecting disruption budgets) but complicated maintennce workflow scheduling shouldn't be part of core Kubernetes.

Is this covered by kubectl drain?

Where are we on Docker checkpoint and restore? It would be super ideal if we could combine the auto drain with a pod migration procedure.

kubectl drain does a lot of this, but there are some ideas in this issue that were not really ever implemented and we might want to refer back to.

I am closing this as fixed. We can easily search for and refer to this issue for ideas later.