Failed to list *v1.Pod & v1.service - getsockopt: connection refused err: failed to get node info: node "server1" not foundย #60884
Description
Unable to proceed with kubeadm init due to the log below.
This is just for the initial setup. I've tried:
- Removing the Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" line from the 10-kubelet.conf file
- Copying /etc/kubernetes/admin.conf to $HOME/.kube/config and restarting kubelet + daemon-reload
kubectl apply -f doesn't seem to work as well due to the error below:
The connection to the server 107.105.136.28:6443 was refused - did you specify the right host or port?
What happened:
Please see below log from journalctl -xeu kubelet
Mar 07 20:42:25 server1 kubelet[14619]: E0307 20:42:25.908557 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:26 server1 kubelet[14619]: E0307 20:42:26.718263 14619 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "server1" not found
Mar 07 20:42:26 server1 kubelet[14619]: E0307 20:42:26.906939 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:26 server1 kubelet[14619]: E0307 20:42:26.907933 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:26 server1 kubelet[14619]: E0307 20:42:26.908994 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.392766 14619 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded w
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.392788 14619 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-apiserver-server1_kube-system(7c74aa0f4b9044a62ba3fc2b222b6a49)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request c
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.392795 14619 kuberuntime_manager.go:647] createPodSandbox for pod "kube-apiserver-server1_kube-system(7c74aa0f4b9044a62ba3fc2b222b6a49)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.392823 14619 pod_workers.go:186] Error syncing pod 7c74aa0f4b9044a62ba3fc2b222b6a49 ("kube-apiserver-server1_kube-system(7c74aa0f4b9044a62ba3fc2b222b6a49)"), skipping: failed to "CreatePodSandbox" for "kube-apiserver-server1_kube-system(7c74aa0f4b9044a62ba3fc2b222b6a49)" with CreatePodSandboxError: "CreatePodSandbo
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.907221 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.908220 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.909242 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:28 server1 kubelet[14619]: I0307 20:42:28.239272 14619 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach
Mar 07 20:42:28 server1 kubelet[14619]: I0307 20:42:28.246581 14619 kubelet_node_status.go:82] Attempting to register node server1
Mar 07 20:42:28 server1 kubelet[14619]: E0307 20:42:28.246744 14619 kubelet_node_status.go:106] Unable to register node "server1" with API server: Post https://107.105.136.28:6443/api/v1/nodes: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:28 server1 kubelet[14619]: E0307 20:42:28.907491 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:28 server1 kubelet[14619]: E0307 20:42:28.908477 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:28 server1 kubelet[14619]: E0307 20:42:28.909606 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:29 server1 kubelet[14619]: E0307 20:42:29.907804 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:29 server1 kubelet[14619]: E0307 20:42:29.908776 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:29 server1 kubelet[14619]: E0307 20:42:29.909785 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:30 server1 kubelet[14619]: E0307 20:42:30.908131 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:30 server1 kubelet[14619]: E0307 20:42:30.909155 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:30 server1 kubelet[14619]: E0307 20:42:30.910228 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:31 server1 kubelet[14619]: E0307 20:42:31.908495 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:31 server1 kubelet[14619]: E0307 20:42:31.909392 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:31 server1 kubelet[14619]: E0307 20:42:31.910461 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:32 server1 kubelet[14619]: E0307 20:42:32.018611 14619 event.go:209] Unable to write event: 'Patch https://107.105.136.28:6443/api/v1/namespaces/default/events/server1.1519a1c427a76a94: dial tcp 107.105.136.28:6443: getsockopt: connection refused' (may retry after sleeping)
What you expected to happen:
kubernetes should be able to initialize successfully since this is the initial setup using kubeadm
How to reproduce it (as minimally and precisely as possible):
apt-get install -y apt-transport-https
sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
sudo echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee
/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
apt-get update
apt-get install -y kubelet kubeadm kubectl kubernetes-cni
swapoff -a
kubeadm init
Anything else we need to know?:
This was the initial error after kubeadm init. It got stuck for a while at "[init] This might take a minute or longer if the control plane images have to be pulled."
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- There is no internet connection, so the kubelet cannot pull the following control plane images:
- gcr.io/google_containers/kube-apiserver-amd64:v1.9.3
- gcr.io/google_containers/kube-controller-manager-amd64:v1.9.3
- gcr.io/google_containers/kube-scheduler-amd64:v1.9.3
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster
Environment:
-
Kubernetes version (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} -
Cloud provider or hardware configuration:
Local Server (Ubuntu 16.04.3) -
Kernel (e.g.
uname -a
):
Linux server1 4.13.0-36-generic Link fix to cloud consoleย #40~16.04.1-Ubuntu SMP Fri Feb 16 23:25:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux -
Install tools:
kubeadm -
Others: