With a UDP "connection" you can be sending packets to the old IP to your
heart's content and they will go nowhere and that is just part of using
UDP.

Take the proxy out of the picture: If you net.DialUDP to a non-existent
port and send data, it will happily "succeed" but there's nobody
listening. If you have a UDP "connection" to an IP:port and are sending
data and the remote side dies, your app doesn't really know (unless the
protocol you define atop UDP detects it).

I guess we could use conntrack -D -p udp probably with -r (but I'd have
to try it) to kill NAT entries. Seems like a pretty easy project for
someone to tackle.

On Tue, Dec 22, 2015 at 1:32 PM, Joshua Kwan notifications@github.com
wrote:

I encountered a strange issue after opting-in to kube-proxy iptables
support.

Steps to repro (I think):

1. Create a Service for a UDP port (say 8125; in my case, this was statsd)
2. Create pods that bind to this Service
3. Start a client that uses the Service, e.g. via environment variable
   which specifies the VIP for the Service. This client writes a UDP packet to
   the Service every 10 seconds using the same socket (e.g. DialUDP is only
   called once)
4. Log in to the node. Use ngrep -q '' udp port 8125 to view the outgoing
   UDP traffic from the box to the service port. Observe that rewriting has
   occurred, and the packet is destined to one of the concrete endpoints
   specified in the Kubernetes model.
5. Delete that endpoint by, for example, killing the corresponding Pod
   which the client is communicating with
6. Observe that the Pod's IP no longer appears in endpoints (e.g. using
   kubectl)
7. BUG: Observe that ngrep continues to report that the packets are being
   rewritten to the old endpoint.
8. BUG (corollary): Observe that conntrack
   http://conntrack-tools.netfilter.org/ -d SERVICE_VIP shows that same
   socket being routed to the old endpoint.
9. WORKAROUND: Restart the client Pod, or have the client Pod call DialUDP
   each time it needs to send data.

The only solution i can see is that kube-proxy, when it rewrites iptables
rules based on endpoints changes, needs to reset connections between local
sockets and destroyed endpoints.

This didn't happen with the userspace kube-proxy, because kube-proxy was
accepting the packets locally regardless of the endpoints, and would always
use the latest endpoints information to forward the packet on.

Sorry for the long bug report, but I think it should be pretty clear by
now if you've made it here. :)

—
Reply to this email directly or view it on GitHub
#19029.

I guess we could use conntrack -D -p udp probably with -r (but I'd have
to try it) to kill NAT entries. Seems like a pretty easy project for
someone to tackle.

Yeah, this is basically my suggestion here. With the userspace kube-proxy in this scenario, as long as the proxy itself stays up, the endpoints can rotate out without affecting reachability.

With iptables, when an endpoint rotates out, the socket will continue connecting to a dead endpoint. So that's a concrete downside of iptables mode that is pretty hard to figure out without a pretty involved debug session like the one i did for the OP, thus I think kube-proxy should try to do something about it.

Renaming to better reflect issue

Running into this issue with pretty serious consequences for DNS resolution. Some nodes in our cluster needed a reboot to update to a newer CoreOS version. One of the nodes was running a DNS pod. To minimize the effect on services, I first scaled the DNS replication controller to two instances. Then I rebooted the nodes one by one. After the update, I noticed that some services on nodes that didn't need a reboot had trouble resolving the new addresses of services on rebooted nodes. The problem appeared to be related to some stale connection tracking state on the node that routed DNS packets to the wrong/old DNS pod IP.

The result was that services couldn't find the new addresses of pods on rebooted nodes since they were still querying an old DNS pod IP.

Yeah. I'd love to get a patch to handle this. I'm personally buried right now and there's no way I will get to this in the immediate future.

This is a great community project - someone out there must be interested in networking stuff and wants to contribute....

I'll also tag @freehan in case he has cycles, but this is not as high prio as the myriad other things I know you have going on, too.

I have cycles. I can take a look.

Minhan, is this something you're still hoping to look at, or overflowed?

On Mon, Feb 29, 2016 at 11:03 PM, Minhan Xia notifications@github.com
wrote:

I have cycles. I can take a look.

—
Reply to this email directly or view it on GitHub
#19029 (comment)
.

I will submit PR shortly.

oh, fantastic. Way better answer than I expected.

On Thu, Mar 3, 2016 at 5:01 PM, Minhan Xia notifications@github.com wrote:

I will submit PR shortly.

—
Reply to this email directly or view it on GitHub
#19029 (comment)
.

Still happens to me, at least in Node.js. Each time I recreate PODs which are part of UDP service I also have to restart the Node.js PODs.

Using k8s v1.3.6 provisioned by kops

@shamil Can you please open a new issue and post a repro case, as simple as you can make it.

Thanks

@girishkalele @kubernetes/sig-network

@thockin @shamil
Hello guys, sorry for the necro-bump, I think I am facing the same issue (#26309 (comment)), I see that the ticket is closed and @thockin asked @shamil to open a new issue, but I couldn't find any, what is the status? Thank you.