Traffic from GCE LoadBalancer blocked by iptables filter #98
Description
I have kubernets cluster on google cluster. One master and one minion. Kubeadm was used to create cluster. Canal(flannel) is used as cni provider. One KubeService was deployed with type LoadBalancer.
Google randomly distributed traffic to master and minion, but master don't accept requests.
nat iptables from master node:
-A KUBE-FW-MX7ZTTA3CLR5PD5H -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -j KUBE-MARK-MASQ
-A KUBE-FW-MX7ZTTA3CLR5PD5H -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -j KUBE-SVC-MX7ZTTA3CLR5PD5H
-A KUBE-FW-MX7ZTTA3CLR5PD5H -s xxx.xxx.xxx.xxx/32 -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -j KUBE-SVC-MX7ZTTA3CLR5PD5H
-A KUBE-FW-MX7ZTTA3CLR5PD5H -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -j KUBE-MARK-DROP
-A KUBE-NODEPORTS -p tcp -m comment --comment "jenkins/jenkins-jenkins:http" -m tcp --dport 30467 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jenkins/jenkins-jenkins:http" -m tcp --dport 30467 -j KUBE-SVC-MX7ZTTA3CLR5PD5H
-A KUBE-SEP-ETTYJLHSYBMK522U -s 10.244.2.16/32 -m comment --comment "jenkins/jenkins-jenkins:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-ETTYJLHSYBMK522U -p tcp -m comment --comment "jenkins/jenkins-jenkins:http" -m tcp -j DNAT --to-destination 10.244.2.16:8080
-A KUBE-SEP-WZT46T3WAUJMMZNP -s 10.244.2.16/32 -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener" -j KUBE-MARK-MASQ
-A KUBE-SEP-WZT46T3WAUJMMZNP -p tcp -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener" -m tcp -j DNAT --to-destination 10.244.2.16:50000
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.0.41.173/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins:http cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.0.41.173/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins:http cluster IP" -m tcp --dport 8080 -j KUBE-SVC-MX7ZTTA3CLR5PD5H
-A KUBE-SERVICES -d xxx.xxx.xxx.xxx/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -m tcp --dport 8080 -j KUBE-FW-MX7ZTTA3CLR5PD5H
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.0.226.250/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener cluster IP" -m tcp --dport 50000 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.0.226.250/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener cluster IP" -m tcp --dport 50000 -j KUBE-SVC-GMUNCQ4ZNKK7N5PD
-A KUBE-SVC-GMUNCQ4ZNKK7N5PD -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener" -j KUBE-SEP-WZT46T3WAUJMMZNP
-A KUBE-SVC-MX7ZTTA3CLR5PD5H -m comment --comment "jenkins/jenkins-jenkins:http" -j KUBE-SEP-ETTYJLHSYBMK522U
filter iptables from master node:
-A FORWARD -m comment --comment "cali:wUHhoiAYhphO9Mso" -j cali-FORWARD
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A cali-FORWARD -i cali+ -m comment --comment "cali:X3vB2lGcBrfkYquC" -j cali-from-wl-dispatch
-A cali-FORWARD -o cali+ -m comment --comment "cali:UtJ9FnhBnFbyQMvU" -j cali-to-wl-dispatch
-A cali-FORWARD -i cali+ -m comment --comment "cali:Tt19HcSdA5YIGSsw" -j ACCEPT
-A cali-FORWARD -o cali+ -m comment --comment "cali:9LzfFCvnpC5_MYXm" -j ACCEPT
-A cali-FORWARD -m comment --comment "cali:7AofLLOqCM5j36rM" -j MARK --set-xmark 0x0/0xe000000
-A cali-FORWARD -m comment --comment "cali:QM1_joSl7tL76Az7" -m mark --mark 0x0/0x1000000 -j cali-from-host-endpoint
-A cali-FORWARD -m comment --comment "cali:C1QSog3bk0AykjAO" -j cali-to-host-endpoint
-A cali-FORWARD -m comment --comment "cali:DmFiPAmzcisqZcvo" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x1000000/0x1000000 -j ACCEPT
Workaround:
I added additional log to iptables:
iptables -A FORWARD -m comment --comment drop -j LOG --log-prefix drop
And found dropped packets:
kernel: [16887.164101] dropIN=ens4 OUT=flannel.1 ......
Final solution that works for me:
iptables -A FORWARD -i flannel+ -j ACCEPT
iptables -A FORWARD -o flannel+ -j ACCEPT
I am not sure that it is right solution.