Indeed, I asked if the AWS ticket could be reopened already, since the current fix makes the entire cluster depend on the availability of a single API. It's extremely dangerous in production.

@mwielgus I think the openstack label here is a little misleading by itself. The issue most likely affects all cloud providers, as evidenced by the symptoms and the other similar issues already opened.

cc/ @gmarek

It may be the time to rethink some parts of NodeController's logic. By the time it was written only source of NotReady Node status was NodeController itself (Kubelet was never writing NotReady), but those times are long gone and I think that we might act slightly differently when NotReady was set by NodeController (machine is not responsive) and when it was set by Kubelet itself (some software problem - CloudProvider is not responsive, Docker daemon has some problems, etc.).

Current logic of NodeController was designed to handle first case for single node failures. As a way to protect us from destroying whole cluster in case of networking problems of the master machine rate limiting was introduces (@brendandburns is right, that we should test this). Current limits were set at-hock in a world of 100 Node clusters - this certainly needs a fix I'll open an issue for it.

As for the main problem: I think that we should distinguish single(-ish) Node failures from whole-cluster ones. When all Nodes are NotReady ControllerManager can assume that it's some kind of global problem (e.g. cloud provider is not responsive) or that master machine is separated from the cluster. In both those cases it shouldn't take any action, as it won't do any good. One of the question is how to proceed when Nodes will start to reappear gradually. Maybe we should have separate 'healing' state of NodeController, or maybe it can be simple incorporated into the normal logic.

An orthogonal issue is if we wan't to distinguish 'hard' failures (NotReady set by the NodeController) from 'soft' ones (NotReady set by the Kubelet). We may want to give Kubelet additional time to try to fix itself (e.g. by restarting Docker or sth), but I'm not sure it's worth the effort.

Action items:

• provide better limits for eviction rate limiter,
• write a test for eviction rate limiting (I don't think it needs to be an e2e test, I think that it may be possible to write a unit test for it, as it doesn't need any component other than NodeControler),
• add a logic for handling 'whole cluster down' situation to NodeController and make sure that recovery is possible
• decide if we want to act differently on Kubelet-driven node's problems.

It's not possible to all of this for 1.2 (i.e. until tomorrow;)

cc @davidopp @bgrant0607 @wojtek-t @fgrzadkowski @smarterclayton @quinton-hoole

The more data we have, the better decisions we can make.

It's useful to distinguish VM is unreachable vs. Kubelet not communicating.

It's useful to distinguish Kubelet can't run any containers vs. can't start new ones.

It's useful to know why: Docker dead/unresponsive, sys OOM, OOD, volumes can't be mounted, container network unavailable, etc.

It's useful to determine which failures are correlated: all, by fault domain, with software rollout, particular dedicated pools, with a Container of Death, etc.

In response, we might want to make the node unschedulable, evict scheduled pods, reboot the node, replace the node, revert the rollout, etc.

Rate limiting is just a way to slow down to human speeds in hope of human analysis or intervention. It's necessary in some cases, but we could do better in others.

Fixed by #25571