@derekwaynecarr There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-* for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

/cc @kubernetes/sig-node-bugs @sjenning @vishh

cc @wojtek-t @gmarek

we have to setup some timeout, but i am worried about the potential impact on any watch code.

note: this is different than the Dial timeout (which we do appear to have default).

I remember @jayunit100 has a related PR but I don't know if it has been merged finally.

need to take a pass at kube-proxy too. i see no explicit timeout there as well at first glance.

@derekwaynecarr fyi I reported this earlier as #44557

#48926 is a blocker for this.

@obeattie had a recommendation worth evaluating here:
#41916 (comment)

it would require us to tune net.ipv4.tcp_retries2
see: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

in practice, it would make sense for this value to potentially be monitored by the kubelet to ensure node status update interval falls within desired range.

@vishh @gmarek - what are your thoughts? longer term, we should still split client timeouts for long vs short operations, but this tweak would help address many situations where LB communication to a master introduces problems.

@derekwaynecarr Have you seen #48670?

@obeattie will take a look, looks promising.

We've had three major events in the last few weeks that comes down to this problem. Watches set up through an elb node that gets replaced or scaled down cause large numbers of nodes to go not ready for 15 minutes causing very scary cluster turbulence. ( We've generally seen between a third to half the nodes go not ready ). We're currently evaluating other ways to load balance the api servers for the components we currently send through the elb ( I haven't poured through everything but I think that boils down to the kubelet and the proxy (possibly flannel) ).

I know it's not strictly related to this problem, but after a third of a 'zone' goes down NodeController will drastically reduce ratio in which it evicts Pods - exactly to protect you from screwing your cluster too much in situations like this.

Given that it is important problem you could try pushing @kubernetes/sig-api-machinery-feature-requests to make required changes to client-go

/sig azure
Added for visibility

reopening, the hang is resolved but the underlying stuck TCP connection issue is not

since we're already tracking open API connections from the kubelet in client cert rotation cases, and have the ability to force close those connections, the simplest change is likely to be to reuse that option in response to a heartbeat failure. that has the added benefit of dropping dead connections for all kubelet -> API calls, not just the heartbeat client

WIP in #63492

I am wondering if this would also apply to the kube-proxy. As it also maintains a “connection” to the api server and would suffer from the same (???).

Ideally fix should be in client-go

@recollir yes, I'm pretty sure you're right. This issue should be rescoped to include kube-proxy IMO

one issue at a time :)

persistent kubelet heartbeat failure results in all workloads being evicted. kube-proxy network issues are disruptive for some workloads, but not necessarily all

kube-proxy (and general client-go support) would need a different mechanism, since those components do not heartbeat with the api like the kubelet does. I'd recommend spawning a separate issue for kube-proxy handling of this condition.

Since this issue has been re-opened, would there be any value in me re-opening my PR for this commit? Monzo has been running this patch in production since last July and it has eliminated this problem entirely, for all uses of client-go.

Since this issue has been re-opened, would there be any value in me re-opening my PR for this commit? Monzo has been running this patch in production since last July and it has eliminated this problem entirely, for all uses of client-go.

I still have several reservations about that fix:

• it relies on behavior that appears to be undefined (it changes properties on the result of net.TCPConn#File(), documented as "The returned os.File's file descriptor is different from the connection's. Attempting to change properties of the original using this duplicate may or may not have the desired effect.")
• calling net.TCPConn#File() has implications I'm unsure of: "On Unix systems this will cause the SetDeadline methods to stop working."
• it appears to only trigger closing in response to unacknowledged outgoing data... that doesn't seem like it would help clients (like kube-proxy) with long-running receive-only watch stream connections

that said, if @kubernetes/sig-api-machinery-pr-reviews and/or @kubernetes/sig-network-pr-reviews feel strongly that is the correct direction to pursue, that would be helpful to hear.

Few notes on these very valid concerns:

• https://golang.org/pkg/net/#TCPConn.File is returning dup'ed filedescriptor, which AFAIK share all underneath structures in kernel, except for entry in file descriptor table, so either can be used with same results. Program should be aware not to try to use them simultaneously though, for exact same reasons.
• today returned filedescriptor is set to blocking mode. Probably it can be mitigated by setting it back to nonblocking mode. In Go 1.11 returned fd is going to be in same blocking/unblocking mode as it was before .File() call: net: File method of {TCP,UDP,IP,Unix}Conn and {TCP,Unix}Listener should leave the socket in nonblocking mode golang/go#24942
• Maybe it will not help simple watchers, I am not familiar with Informers internals, but I was under the impression that they are not only watching, but also periodically resyncing state, these resync would trigger outgoing data transfer which would then be detected .

Indeed: as far as I understand, the behaviour is not undefined, it's just defined in Linux rather than in Go. I think the Go docs could be clearer on this. Here's the relevant section from dup(2):

After a successful return from one of these system calls, the old and new file descriptors may be used interchangeably. They refer to the same open file description (see open(2)) and thus share file offset and file status flags; for example, if the file offset is modified by using lseek(2) on one of the descriptors, the offset is also changed for the other.

The two descriptors do not share file descriptor flags (the close-on-exec flag).

My code doesn't modify flags after obtaining the fd, instead its only use is in a call to setsockopt(2). The docs for that call are fairly clear that it modifies properties of the socket referred to by the descriptor, not the descriptor itself:

getsockopt() and setsockopt() manipulate options for the socket referred to by the file descriptor sockfd.

I agree that the original descriptor being set to blocking mode is annoying. Go's code is clear that this will not prevent anything from working, just that more OS threads may be required for I/O:

https://github.com/golang/go/blob/516f5ccf57560ed402cdae28a36e1dc9e81444c3/src/net/fd_unix.go#L313-L315

Given that a single Kubelet (or otherwise use of client-go) establishes a small number of long-lived connections to the apiservers, and that this will be fixed in Go 1.11, I don't think this is a significant issue.

I am happy for this to be fixed in another way, but given we know that this works and does not require invasive changes to the apiserver to achieve, I think it is a reasonable solution. I have heard from several production users of Kubernetes that this has bitten them in the same way it bit us.

this issue is resolved for the kubelet in #63492

#65012 is open for the broader client-go issue. hoisted the last few comments to that issue

/close

Did scheduler & controller → api-server has the same issue?

@workhardcc , yes, both use client-go , which AFAIK remains unfixed

@redbaron Did scheduler & controller fix this problem？ If not, kubelet connect api-server ok（reconnect）, but scheduler & controller connect api-server failed. This is also an exception.

I'm experiencing this again with k8s 1.13.4 and azure only (filtered with kubelet_node_status.go):

I0319 17:14:25.579175   68746 kubelet_node_status.go:478] Setting node status at position 6
I0319 17:29:29.987934   68746 kubelet_node_status.go:478] Setting node status at position 7

For 15 minutes node doesn't update status. There are no issues on network level and node actually is fully operational. Adjusting --node-monitor-grace-period= on controller-manager helps, but that is not a solution

@liggitt

this issue is resolved for the kubelet in #63492

I am running into this issue with Kubernetes 1.14.1. It looks like the fix in #63492 is in the kubelet client certificate code. Will this still work if kubelet is using token (not certificate) authentication?

This regressed, and was refixed in 1.14.3

See #78016

@derekwaynecarr , you wrote

we have to setup some timeout, but i am worried about the potential impact on any watch code.

It turns out, there is an impact on watch code, but it doesn't happen with kubemark/hollow nodes, therefore it escaped detection for a long time.

I've tagged you in on the PR I just made to fix the issue, and would appreciate your comments on whether that PR looks reasonable.