Skip to content

Traffic from GCE LoadBalancer blocked by iptables filter #98

Closed
@dreyk

Description

I have kubernets cluster on google cluster. One master and one minion. Kubeadm was used to create cluster. Canal(flannel) is used as cni provider. One KubeService was deployed with type LoadBalancer.
Google randomly distributed traffic to master and minion, but master don't accept requests.

nat iptables from master node:

-A KUBE-FW-MX7ZTTA3CLR5PD5H -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -j KUBE-MARK-MASQ
-A KUBE-FW-MX7ZTTA3CLR5PD5H -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -j KUBE-SVC-MX7ZTTA3CLR5PD5H
-A KUBE-FW-MX7ZTTA3CLR5PD5H -s xxx.xxx.xxx.xxx/32 -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -j KUBE-SVC-MX7ZTTA3CLR5PD5H
-A KUBE-FW-MX7ZTTA3CLR5PD5H -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -j KUBE-MARK-DROP
-A KUBE-NODEPORTS -p tcp -m comment --comment "jenkins/jenkins-jenkins:http" -m tcp --dport 30467 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jenkins/jenkins-jenkins:http" -m tcp --dport 30467 -j KUBE-SVC-MX7ZTTA3CLR5PD5H
-A KUBE-SEP-ETTYJLHSYBMK522U -s 10.244.2.16/32 -m comment --comment "jenkins/jenkins-jenkins:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-ETTYJLHSYBMK522U -p tcp -m comment --comment "jenkins/jenkins-jenkins:http" -m tcp -j DNAT --to-destination 10.244.2.16:8080
-A KUBE-SEP-WZT46T3WAUJMMZNP -s 10.244.2.16/32 -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener" -j KUBE-MARK-MASQ
-A KUBE-SEP-WZT46T3WAUJMMZNP -p tcp -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener" -m tcp -j DNAT --to-destination 10.244.2.16:50000
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.0.41.173/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins:http cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.0.41.173/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins:http cluster IP" -m tcp --dport 8080 -j KUBE-SVC-MX7ZTTA3CLR5PD5H
-A KUBE-SERVICES -d xxx.xxx.xxx.xxx/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins:http loadbalancer IP" -m tcp --dport 8080 -j KUBE-FW-MX7ZTTA3CLR5PD5H
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.0.226.250/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener cluster IP" -m tcp --dport 50000 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.0.226.250/32 -p tcp -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener cluster IP" -m tcp --dport 50000 -j KUBE-SVC-GMUNCQ4ZNKK7N5PD
-A KUBE-SVC-GMUNCQ4ZNKK7N5PD -m comment --comment "jenkins/jenkins-jenkins-agent:slavelistener" -j KUBE-SEP-WZT46T3WAUJMMZNP
-A KUBE-SVC-MX7ZTTA3CLR5PD5H -m comment --comment "jenkins/jenkins-jenkins:http" -j KUBE-SEP-ETTYJLHSYBMK522U

filter iptables from master node:

-A FORWARD -m comment --comment "cali:wUHhoiAYhphO9Mso" -j cali-FORWARD
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A cali-FORWARD -i cali+ -m comment --comment "cali:X3vB2lGcBrfkYquC" -j cali-from-wl-dispatch
-A cali-FORWARD -o cali+ -m comment --comment "cali:UtJ9FnhBnFbyQMvU" -j cali-to-wl-dispatch
-A cali-FORWARD -i cali+ -m comment --comment "cali:Tt19HcSdA5YIGSsw" -j ACCEPT
-A cali-FORWARD -o cali+ -m comment --comment "cali:9LzfFCvnpC5_MYXm" -j ACCEPT
-A cali-FORWARD -m comment --comment "cali:7AofLLOqCM5j36rM" -j MARK --set-xmark 0x0/0xe000000
-A cali-FORWARD -m comment --comment "cali:QM1_joSl7tL76Az7" -m mark --mark 0x0/0x1000000 -j cali-from-host-endpoint
-A cali-FORWARD -m comment --comment "cali:C1QSog3bk0AykjAO" -j cali-to-host-endpoint
-A cali-FORWARD -m comment --comment "cali:DmFiPAmzcisqZcvo" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x1000000/0x1000000 -j ACCEPT

Workaround:
I added additional log to iptables:
iptables -A FORWARD -m comment --comment drop -j LOG --log-prefix drop
And found dropped packets:
kernel: [16887.164101] dropIN=ens4 OUT=flannel.1 ......

Final solution that works for me:
iptables -A FORWARD -i flannel+ -j ACCEPT
iptables -A FORWARD -o flannel+ -j ACCEPT

I am not sure that it is right solution.

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions