related issues:
#3333
#2524
#1573

@zmerlynn

good to see a familiar face, @multilinear

Contribs are always welcome. Things change rapidly, so the best approach is to break it up and contribute incrementally.

We've been assuming that we have to respect an SLA for pods to do node upgrades. If for your cluster this is not necessary then you could script this yourself without any changes to the kubernetes code.

I don't think you need to add API functionality to do this.
Hints:

• switch from master-based node management to "manual" node management. See the bottom of docs/node.md talks about this. --sync_nodes=false
• If you kubectl remove nodes $nodename, then it won't be schedulable. But the pods will keep running for a bit. Then you could gracefully stop all those pods within 5 minutes before they are ungracefully killed.

Let's definitely sync. @maxforbes on our side has already started looking at how to do rolling updates for GKE, and it's similar to what you're saying. We were thinking of starting with simple drain and re-provision as well.

I'm considering more grandiose plans to possibly rewrite kube-up.sh and friends to take a yaml specification so we can be a lot more precise for provisioning and update. I expect to have a proposal out soon, assuming I get some time for that this week. (I think I can take all the existing scripts along for the ride and work towards rewriting advanced providers in Go as we want.) So that's something else in this puzzle to think about.

Silly mobile client. I meant @mbforbes

I see, so you can do this via "kubectl create" and "kubectl delete". Then indeed manual management would definitely work for now. There's a bit more already there than I realized... I'll have to give that a try. Thanks!

It sounds like zmerlynn has more thoughts though, and I'm still interested in solutions that build a bit more into the kubernetes framework as well (be it shell scripts or otherwise). So I'll leave this open.

@multilinear can you report back and let us know if the manual strategy worked and/or if you hit any bumps?

Also, once #4585 merges, which should be soon, there will be an explicit unschedulable-but-not-deleted state for nodes. That will give you as much time as you want to drain the node.

With the round-up for node upgrades (#6079), I'm trying to close other issues that target pieces of the process. Specifically, this should be implemented with the basic version (#6082).

Your ideas here were definitely helpful, though! I think they're going to be taking shape in the system:

• API functionality to add & remove nodes will be in pkg/cloudprovider
• the tool / libraries to add & remove nodes this will be kubedeploy (Proposal to rework Kubernetes deployment CLI #5472)
• doing this dynamically is a super important issue (Allow nodes to dynamically join a running cluster #6087, part of Secure node -> master communication #3168)
• rolling update should be the basic mechanism for upgrades (Node upgrades: basic mechanism #6082)
• you clearly wanted the mechanisms by which this is done to be in Go in a non-horrible way; we want a step more and have the top-level mechanism to be Go rather than a Bash script (Node upgrades: write in Go and hook into pkg/cloudprovider #6095)
• and you were doing this on AWS; we also want a provider-agnostic solution (Node upgrades: provider agnosticism #6096)

Definitely re-open / ping if I missed something!

I see. I don't actually trust the methods in #6082 (basically I don't think it will actually work), and would argue that just recreating the machine form scratch is much smarter. Truly idempotent machine setup is nigh impossible (Chef doesn't do it, and that's their entire product), and I'm not sure it has much value if you're running inside a virtualized cluster anyway... but I suppose that's an argument to have on that bug :).

So, with that sort-of exception, yes I think you hit all the points.

Yeah, believe me, I'd rather start from a fresh machine (huge discussion on this in #3333). However, as Zach mentioned in his reply to you in #6082, we need to be able to let new nodes into the cluster, aka dynamic clustering (#6087), before we can do this. And we want to make some progress before then, as we've been blocked on this for a long time.

As soon as dynamic clustering is in, we have #6088 to start with a fresh machine instead.