Closed
Description
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST
Please provide the following details:
Environment:
minikube version: v0.28.2
OS: MacOS 10.13
VM driver: xhyve
ISO version: ~/.minikube/cache/iso/minikube-v0.28.1.iso
- Install tools:
- Others:
What happened:
kube-proxy won't start in ipvs mode
What you expected to happen:
kube-proxy should start in ipvs mode
How to reproduce it (as minimally and precisely as possible):
After minikube boots, set mode: "ipvs" in config.conf in kube-proxy ConfigMap:
kubectl edit -n kube-system configmap/kube-proxyTo apply new configuration, delete the old pod and k8s will create a new one,
as required by corresponding DaemonSet:
$ kc get -n kube-system pods
NAME READY STATUS RESTARTS AGE
...
kube-proxy-49psk 1/1 Running 0 11h
...
$ kc delete -n kube-system po/kube-proxy-49psk
pod "kube-proxy-49psk" deleted
$ kc get -n kube-system pods
NAME READY STATUS RESTARTS AGE
...
kube-proxy-x7qgq 1/1 Running 0 7m
...Output of minikube logs (if applicable):
$ kc logs -n kube-system po/kube-proxy-x7qgq
E0805 09:46:12.625751 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-CLUSTER-IP hash:ip,port inet 1024 65536 0-65535 Kubernetes service cluster ip + port for masquerade purpose} map[] 0xc420562080} exist, error: error creating ipset KUBE-CLUSTER-IP, error: exit status 1
E0805 09:46:42.645604 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOAD-BALANCER-FW hash:ip,port inet 1024 65536 0-65535 Kubernetes service load balancer ip + port for load balancer with sourceRange} map[] 0xc420562080} exist, error: error creating ipset KUBE-LOAD-BALANCER-FW, error: exit status 1
E0805 09:47:12.677159 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-NODE-PORT-UDP bitmap:port inet 1024 65536 0-65535 Kubernetes nodeport UDP port for masquerade purpose} map[] 0xc420562080} exist, error: error creating ipset KUBE-NODE-PORT-UDP, error: exit status 1
E0805 09:47:42.748946 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-NODE-PORT-LOCAL-TCP bitmap:port inet 1024 65536 0-65535 Kubernetes nodeport TCP port with externalTrafficPolicy=local} map[] 0xc420562080} exist, error: error creating ipset KUBE-NODE-PORT-LOCAL-TCP, error: exit status 1
Anything else do we need to know:
With all ipset-related modules, kube-proxy starts as expected:
I0811 21:26:07.996804 1 feature_gate.go:230] feature gates: &{map[]}
I0811 21:26:08.064640 1 server_others.go:183] Using ipvs Proxier.
W0811 21:26:08.086817 1 proxier.go:349] clusterCIDR not specified, unable to distinguish between internal and external traffic
W0811 21:26:08.086847 1 proxier.go:355] IPVS scheduler not specified, use rr by default
I0811 21:26:08.087178 1 server_others.go:210] Tearing down inactive rules.
I0811 21:26:08.142232 1 server.go:448] Version: v1.11.0
I0811 21:26:08.154958 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0811 21:26:08.155260 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0811 21:26:08.155338 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0811 21:26:08.155394 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0811 21:26:08.155634 1 config.go:102] Starting endpoints config controller
I0811 21:26:08.155661 1 controller_utils.go:1025] Waiting for caches to sync for endpoints config controller
I0811 21:26:08.155703 1 config.go:202] Starting service config controller
I0811 21:26:08.155709 1 controller_utils.go:1025] Waiting for caches to sync for service config controller
I0811 21:26:08.256254 1 controller_utils.go:1032] Caches are synced for service config controller
I0811 21:26:08.256369 1 controller_utils.go:1032] Caches are synced for endpoints config controller
ipsets, created by kube-proxy:
$ kc exec -it kube-proxy-lxj2d ipset list | grep Type | sort -u
Type: bitmap:port
Type: hash:ip,port
Type: hash:ip,port,ip
Type: hash:ip,port,netSo, the proposed changes are:
diff --git a/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig b/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig
index e5de73c4d..bb860f22e 100644
--- a/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig
+++ b/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig
@@ -187,7 +187,11 @@ CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
CONFIG_IP_SET=y
+CONFIG_IP_SET_BITMAP_PORT=m
CONFIG_IP_SET_HASH_IP=m
+CONFIG_IP_SET_HASH_IPPORT=m
+CONFIG_IP_SET_HASH_IPPORTIP=m
+CONFIG_IP_SET_HASH_IPPORTNET=m
CONFIG_IP_SET_HASH_NET=m
CONFIG_IP_SET_LIST_SET=m
CONFIG_IP_VS=m