Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

(arguably WAI, but I have thoughts about that in the "what you expected" section)

What happened:

On my 1.10.5 cluster (kubeadm, deb package for kubelet), in preparation for upgrading to 1.11.0, I accidentally upgraded the kubelets before the control plane (apt update && apt upgrade with kubelet version not pinned, results in kubelet+kubeadm 1.11.0 vs. 1.10.5 control plane).

After the kubelet upgrade was completed, all kubelets in my cluster were crashlooping due to a missing /var/lib/kubelet/config.yaml. This means the control plane was also hard down.

I couldn't execute kubeadm upgrade apply v1.11.0 to fix the version skew, because it needs the control plane to be up.

I tried downgrading kubelet to v1.10.5, but the changes made to the systemd drop-in unit were permanent, so kubelet was still crashlooping after downgrade. This is what I believe to be a bug, the fact that downgrading back to a supported set of component versions did not recover the cluster. In other words, "rollback did not roll back".

The release notes just say that "there needs to be a config.yaml for kubelet now", without specifying what goes in that config, how to construct one (either manually or automatically).

I got lucky, and knew just enough kubeadm to bail myself out. In case someone else with this failure mode finds this bug, the way to fix this that worked for me was the following (I can't promise that it works, but it seems to have worked for me) :

• On control plane node, apt install kubelet=1.10.5, kubeadm alpha phase kubelet config write-to-disk --config=/var/lib/kubelet/config.yaml, then systemctl restart kubelet. This should get kubelet started again, and thus the control plane should come back.
• Still on control node, kubeadm upgrade plan v1.11.0 and kubeadm upgrade apply v1.11.0 to upgrade the control plane.
• On damaged worker nodes, run kubeadm alpha phase kubelet config download to generate the config.yaml from the in-cluster ConfigMap (which is now reachable since you repaired the control plane). You might need a systemctl restart kubelet to defeat the crashloop backoff timer.
• Cleanup on all nodes, get all kubelets up to 1.11.0.

What you expected to happen:

I expected Kubernetes to be robust against accidental version skew and out-of-order upgrading. Especially given the current state of managing k8s clusters, the situation where you end up upgrading the kubelet before the control plane is unfortunately common.

By "robust", I don't necessarily mean "should work", since this configuration is clearly outside the supported version skew. However, I think it should not persistently break the cluster. In other words, the only necessary corrective action should have been apt install kubelet=1.10.5 to downgrade back to a supported {control plane}x{kubelet} version set.

At minimum, I expected the Debian package to set things up such that a downgrade goes back to a working configuration. Even better would be if kubelet 1.11 had crashlooped with a useful message, e.g. "oh dear, it looks like you did upgrades out of order, because I don't have a config. Downgrade back to 1.10 and go upgrade the control plane first!" That way, people who don't have a rollback reflex as sharp as mine are still guided in the right direction.

How to reproduce it (as minimally and precisely as possible):

On Debian testing (or ubuntu xenial, failure mode should be identical):

• Add the Docker and Kubernetes apt sources, per official documentation.
• apt install kubelet=1.10.5-00 kubeadm
• kubeadm init on control plane, kubeadm join on worker nodes as appropriate.
• Verify kubectl get nodes works, lists all nodes at 1.10.5.
• apt install kubelet to upgrade kubelet to 1.11.0-00.
• Observe kubectl get nodes can no longer connect to control plane. journalctl -u kubelet shows kubelet crashlooping due to lack of config.yaml.
• apt install kubelet=1.10.5-00 to downgrade kubelet.
• Observe kubectl get nodes still broken, journalctl -u kubelet shows same crashloop.
• Apply manual fixing with individual kubeadm steps, per above
• Observe (hopefully!) that kubectl get nodes works again, and kubelets have started back up.

Environment:

• Kubernetes version (use kubectl version): v1.10.5 -> v1.11.0
• Cloud provider or hardware configuration: kubeadm on debian testing (~= ubuntu xenial for k8s binaries) on bare metal.
• OS (e.g. from /etc/os-release): Debian testing (buster)
• Kernel (e.g. uname -a): Linux prod-01 4.16.0-2-amd64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Debian 4.16.12-1 (2018-05-27) x86_64 GNU/Linux
• Install tools: kubeadm