nodecontroller never finishes updates on 2k nodes #26211
Comments
zmerlynn
added
priority/important-soon
|
@dchen1107: Can you find someone to maybe look at this or triage further? |
|
@Random-Liu Can you take a look at this to make sure the issue is not caused by NodeProblemDetector updating a new NodeCondition? |
|
Sure! |
|
@dchen1107 Normally node-problem-detector should not update node object so frequently. |
|
@dchen1107 I checked the cluster. There is no NodeProblemDetector running in this cluster. :) |
|
output of kubectl describe nodes? |
|
Here is the result of We can see that the transition time of Ready and OutOfDisk is around Then in controller log, the first race happens at Last one happens at |
|
@zmerlynn Can we have related Kubelet logs? |
|
I gave @Random-Liu access. |
|
Looks like we rule out two potential causes based on above output: 1) NodeProblemDetector 2) Out-of-Resource condition updates. Looks like this is specific failure caused by those hollow nodes. cc/ @wojtek-t |
|
What do you mean by hollow nodes? |
|
(This is a real 2k node cluster.) |
|
@zmerlynn Sorry, I misunderstood what you said. I thought you were saying "kubernetes e2e scalability test" |
|
@zmerlynn This doesn't look like a node team problem, this is a nodecontroller problem. nodecontroller is owned by control plane. node controller clearly needs to do something in a retry loop. @mikedanese @gmarek <--- people who know something about node controller. |
|
@Random-Liu the only way this is your bug is if your component does many many writes to the node status on startup. |
|
@lavalamp - I took a fast stab and asked for triage, sorry. To be fair, there's about 2k nodes posting status, but yes, that's probably the controller falling over, not the node team's fault. |
|
But IIUC, this should be fixed by #26207 . Or am I missing something? |
@lavalamp There is no node-problem-detector running in this cluster, but as you said, node-problem-detector should also avoid updating node status at the same time when the cluster start up. @zmerlynn I checked the cluster yesterday. It's wired that all nodes started at 20:12:00, but the apiserver started at 21:27:00. After apiserver started, all the nodes were trying to update node statuses at the same time, and the nodecontroller also started complaining the update error at 21:29:15. After 21:51:11, there is no such error in nodecontroller any more.
It looks like #26207 doesn't solve this one. This one is a real update failure, not incorrect log. :) |
|
@Random-Liu: That time delta is really odd. The master is sometimes slow to create compared to the individual IGMs on GKE, but not the nodes themselves, which shouldn't even be that uniform. |
|
I am suspicious of the time delta-- is one of those from a source that On Wed, May 25, 2016 at 10:57 AM, Zach Loafman notifications@github.com
|
The time on the node: The time on the master: The creation timestamp of node The first log in kube-apiserver.log on the master |
|
Is this kubernetes/node-problem-detector#9 ? |
|
@davidopp Nope, node-problem-detector is not running in the cluster. |
|
I'm giving this to @davidopp to delegate-- I can't see any reason why it should be @Random-Liu's problem. :) |
|
@gmarek do you think you could take a look at this? |
|
I'll look into it on Monday. |
|
Thanks! |
|
I have logs from the last enormous-cluster run. I expect that we're throttling something at a client side, but I need to confirm. |
|
Though I see the problem in route-controller not node-controller itself. |
|
This was observed 2 weeks a go. We did a bunch of changes especially in route-controller since then. Maybe this is no longer an issue. Did we check it? |
|
I'm looking at the last enormous cluster run from June 3rd. |
|
I looked at api-server logs and it looks that it's working as intended (more or less) - conflicts that I was able to find was caused by StatusUpdates sent by Kubelets. This shouldn't be too surprising in 2k Node clusters. @zmerlynn - what was the result of this behavior? NodeStatuses weren't up to date? |
|
At the time, it looked like the nodecontroller was flapping and just not getting around to updating node state. I'm happy to close this for now, though. We've certainly changed enough and I also haven't seen it. |
My cluster has been up for 20m or so and the
nodecontrolleris going crazy (this is filtering out "observed a new Node" messages, because they're very spammy):The text was updated successfully, but these errors were encountered: