Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KubeProxy does not delete connections for traffic to NodePort Service if the backend changes. #42473

Closed
hikhvar opened this issue Mar 3, 2017 · 4 comments
Closed

KubeProxy does not delete connections for traffic to NodePort Service if the backend changes. #42473

hikhvar opened this issue Mar 3, 2017 · 4 comments
Labels
area/kube-proxy sig/network Categorizes an issue or PR as relevant to SIG Network.

Comments

@hikhvar
Copy link

hikhvar commented Mar 3, 2017

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

no

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

nodeport udp

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: own Hardware, kube proxy in iptables mode.
  • OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
  • Kernel (e.g. uname -a):
Linux lej01-i1-srv-08 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others:

What happened:
I setup a service with type NodePort. This service should forward UDP syslog to a logstash pod. The traffic comes from a nginx outside of the cluster. If the backend pod get deleted old connections are not deleted. The conntrack output later shows active connections for both, the old and the new backend.

Service setup:

apiVersion: v1
kind: Service
metadata:
  name: logstash
  labels:
    app: logstash
spec:
  type: NodePort
  selector:
    app: logstash
  ports:
  - name: logstash-nginx-syslog
    port: 5141
    protocol: UDP
    nodePort: 30514
    targetPort: 5141

IP of the old backend: 10.194.3.19
IP of the new backend: 10.194.2.37
IP of the Kubernetes Worker Node: 10.200.20.2

sudo conntrack -L | grep 30514
udp      17 29 src=10.200.12.83 dst=10.200.20.2 sport=43542 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=43542 mark=0 use=1
udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=57669 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=57669 mark=0 use=1
udp      17 29 src=10.200.12.115 dst=10.200.20.2 sport=54444 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=54444 mark=0 use=1
udp      17 29 src=10.200.12.115 dst=10.200.20.2 sport=35148 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=35148 mark=0 use=1
udp      17 23 src=10.200.12.115 dst=10.200.20.2 sport=40428 dport=30514 [UNREPLIED] src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=40428 mark=0 use=1
udp      17 29 src=10.200.12.83 dst=10.200.20.2 sport=49538 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=49538 mark=0 use=1
udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=39479 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=39479 mark=0 use=1
udp      17 29 src=10.200.12.147 dst=10.200.20.2 sport=47248 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=47248 mark=0 use=1
udp      17 27 src=10.200.12.115 dst=10.200.20.2 sport=59714 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=59714 mark=0 use=1
udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=58922 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=58922 mark=0 use=1
udp      17 28 src=10.200.12.67 dst=10.200.20.2 sport=39006 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=39006 mark=0 use=1
udp      17 29 src=10.200.12.131 dst=10.200.20.2 sport=57143 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=57143 mark=0 use=1
udp      17 29 src=10.200.12.99 dst=10.200.20.2 sport=48913 dport=30514 [UNREPLIED] src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=48913 mark=0 use=1
udp      17 29 src=10.200.12.147 dst=10.200.20.2 sport=57293 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=57293 mark=0 use=1
udp      17 28 src=10.200.12.99 dst=10.200.20.2 sport=59793 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=59793 mark=0 use=1
udp      17 29 src=10.200.12.115 dst=10.200.20.2 sport=41956 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=41956 mark=0 use=1
udp      17 0 src=10.200.12.67 dst=10.200.20.2 sport=37252 dport=30514 [UNREPLIED] src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=37252 mark=0 use=1
udp      17 24 src=10.200.12.115 dst=10.200.20.2 sport=49765 dport=30514 [UNREPLIED] src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=49765 mark=0 use=1
udp      17 29 src=10.200.12.99 dst=10.200.20.2 sport=60779 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=60779 mark=0 use=1
udp      17 28 src=10.200.12.99 dst=10.200.20.2 sport=51338 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=51338 mark=0 use=1
udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=35126 dport=30514 [UNREPLIED] src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=35126 mark=0 use=1
udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=53102 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=53102 mark=0 use=1
udp      17 28 src=10.200.12.131 dst=10.200.20.2 sport=36563 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=36563 mark=0 use=1
udp      17 29 src=10.200.12.115 dst=10.200.20.2 sport=42914 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=42914 mark=0 use=2
udp      17 25 src=10.200.12.131 dst=10.200.20.2 sport=59050 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=59050 mark=0 use=1
udp      17 29 src=10.200.12.83 dst=10.200.20.2 sport=56343 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=56343 mark=0 use=2
udp      17 29 src=10.200.12.131 dst=10.200.20.2 sport=42781 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=42781 mark=0 use=1
udp      17 25 src=10.200.12.131 dst=10.200.20.2 sport=52930 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=52930 mark=0 use=1
udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=43209 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=43209 mark=0 use=1
udp      17 29 src=10.200.12.83 dst=10.200.20.2 sport=45244 dport=30514 [UNREPLIED] src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=45244 mark=0 use=1

What you expected to happen:
The connections for the old backend get deleted.

How to reproduce it (as minimally and precisely as possible):

  1. Start a service from with NodePort and UDP.
  2. Add one backend to that service
  3. Produce Traffic to the NodePort
  4. Delete the backend and create new backend.
  5. Watch the connection via conntrack.

Anything else we need to know:

The issue seems to be related to this function: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L791
It is changed in recent versions but the conntrack statement includes the service IP. Since we talk to the NodePort, the service IP does not appear in the conntrack connections. My proposal is to delete all udp connections to the backend IP (endpointIP) and the backendPort.
@thockin thockin added sig/network Categorizes an issue or PR as relevant to SIG Network. and removed sig/network Categorizes an issue or PR as relevant to SIG Network. labels May 16, 2017
@scrwr
Copy link

scrwr commented Jun 16, 2017

Still existing with
$ kube-proxy --version Kubernetes v1.6.4

@cmluciano
Copy link

We now remove the UDP connections when they are stale.

@redbaron
Copy link
Contributor

redbaron commented Dec 6, 2017

@cmluciano would you mind sharing PR with relevant changes, please?

@cmluciano
Copy link

#22573

#32561

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/kube-proxy sig/network Categorizes an issue or PR as relevant to SIG Network.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@redbaron @hikhvar @cmluciano @thockin @dcbw @scrwr