@fgrzadkowski, @vmarmol

Taking a look, thanks for reporting @fabioy!

@fabioy these look like node statuses not pod statuses?

You're probably right. I couldn't readily tell from the logs.

On Tue, Mar 24, 2015 at 1:14 PM, Victor Marmol notifications@github.com
wrote:

@fabioy https://github.com/fabioy these look like node statuses not pod
statuses?

—
Reply to this email directly or view it on GitHub
#5864 (comment)
.

-- Fabio Yeon

Load-balancing to @dchen1107 :) assigning to you

Found the bug in the code, will sent out fix shortly.

It turns out this is no bug, and it works as intended. Under current design, we treat nodeStatus as heatbeat message:

When kubelet first starts up, it post nodeStatus every 500 millisec for first 2sec for a faster cluster startup. After this, kubelet post nodeStatus every 2sec. NodeController processes those heatbeat messages, and if it missed sequentially continuous 4 heatbeats, it will mark unreachable.

Above intervals is reasonable to me, but we could tune them though. I am going to close the issue.

cc/ @ddysher @bgrant0607

As an aside, TCP keepalives are a very cheap way to do these sorts of remote heartbeats (all in-kernel). Then you can potentially do low latency edge-triggered rather than level triggered event reporting.

@quinton-hoole I raised the same point when first introducing nodeStatus as heatbeat. If I remembered it correctly, the answer is that we can evolve it later. @ddysher Correct me if I were wrong here.

Based on the log timing, it's just for faster startup, like @dchen1107 said.

As to performance, yes, we wanted to evolve it later (which is 'now').

Is there another issue tracking the reduction of these PUTs?

Just as a heads up, I started up a completely idle 0.14.2 500 node cluster with an n1-standard-8 instance as the master and the master is completely 100% loaded to the point that 50% of requests return 429 due to node status GET's and PUT's. I know the target 1.0 size is 100 nodes but it would be great to optimize this at least a bit.

@wojtek-t @fgrzadkowski

In my opinion, the first thing that we should do is to increase the "heartbeat duration".
IIUC, the main reason for it is to move the pods that are running on broken machine to another machine in case the machine is unreachable. But I think this case is pretty similar to restarting the pod that failed (e.g. due to crash). However, we are currently examining pods only every 10 seconds. So for me it doesn't give much value to send heartbeat from a node every 2 seconds.
But I also agree that we should probably come up with a better mechanism than sending the whole NodeStatus from Kubelet as a heartbeat.

The issue was filed because we believe there is a bug, but turns out it is config parameters of interval at startup time. To tune the NodeStatus interval and other NodeStatus related performance issues are covered by #5953 and several other issues.

@ghodss Let's move the discussion to #5953.