Skip to content

Failed to list *v1.Pod & v1.service - getsockopt: connection refused err: failed to get node info: node "server1" not foundย #60884

Closed
@SamSinging

Description

Unable to proceed with kubeadm init due to the log below.
This is just for the initial setup. I've tried:

  • Removing the Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" line from the 10-kubelet.conf file
  • Copying /etc/kubernetes/admin.conf to $HOME/.kube/config and restarting kubelet + daemon-reload

kubectl apply -f doesn't seem to work as well due to the error below:
The connection to the server 107.105.136.28:6443 was refused - did you specify the right host or port?

What happened:
Please see below log from journalctl -xeu kubelet

Mar 07 20:42:25 server1 kubelet[14619]: E0307 20:42:25.908557 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:26 server1 kubelet[14619]: E0307 20:42:26.718263 14619 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "server1" not found
Mar 07 20:42:26 server1 kubelet[14619]: E0307 20:42:26.906939 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:26 server1 kubelet[14619]: E0307 20:42:26.907933 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:26 server1 kubelet[14619]: E0307 20:42:26.908994 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.392766 14619 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded w
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.392788 14619 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-apiserver-server1_kube-system(7c74aa0f4b9044a62ba3fc2b222b6a49)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request c
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.392795 14619 kuberuntime_manager.go:647] createPodSandbox for pod "kube-apiserver-server1_kube-system(7c74aa0f4b9044a62ba3fc2b222b6a49)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v2/: net/http: request
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.392823 14619 pod_workers.go:186] Error syncing pod 7c74aa0f4b9044a62ba3fc2b222b6a49 ("kube-apiserver-server1_kube-system(7c74aa0f4b9044a62ba3fc2b222b6a49)"), skipping: failed to "CreatePodSandbox" for "kube-apiserver-server1_kube-system(7c74aa0f4b9044a62ba3fc2b222b6a49)" with CreatePodSandboxError: "CreatePodSandbo
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.907221 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.908220 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:27 server1 kubelet[14619]: E0307 20:42:27.909242 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:28 server1 kubelet[14619]: I0307 20:42:28.239272 14619 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach
Mar 07 20:42:28 server1 kubelet[14619]: I0307 20:42:28.246581 14619 kubelet_node_status.go:82] Attempting to register node server1
Mar 07 20:42:28 server1 kubelet[14619]: E0307 20:42:28.246744 14619 kubelet_node_status.go:106] Unable to register node "server1" with API server: Post https://107.105.136.28:6443/api/v1/nodes: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:28 server1 kubelet[14619]: E0307 20:42:28.907491 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:28 server1 kubelet[14619]: E0307 20:42:28.908477 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:28 server1 kubelet[14619]: E0307 20:42:28.909606 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:29 server1 kubelet[14619]: E0307 20:42:29.907804 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:29 server1 kubelet[14619]: E0307 20:42:29.908776 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:29 server1 kubelet[14619]: E0307 20:42:29.909785 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:30 server1 kubelet[14619]: E0307 20:42:30.908131 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:30 server1 kubelet[14619]: E0307 20:42:30.909155 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:30 server1 kubelet[14619]: E0307 20:42:30.910228 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:31 server1 kubelet[14619]: E0307 20:42:31.908495 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://107.105.136.28:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:31 server1 kubelet[14619]: E0307 20:42:31.909392 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://107.105.136.28:6443/api/v1/nodes?fieldSelector=metadata.name%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:31 server1 kubelet[14619]: E0307 20:42:31.910461 14619 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://107.105.136.28:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dserver1&limit=500&resourceVersion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused
Mar 07 20:42:32 server1 kubelet[14619]: E0307 20:42:32.018611 14619 event.go:209] Unable to write event: 'Patch https://107.105.136.28:6443/api/v1/namespaces/default/events/server1.1519a1c427a76a94: dial tcp 107.105.136.28:6443: getsockopt: connection refused' (may retry after sleeping)

What you expected to happen:
kubernetes should be able to initialize successfully since this is the initial setup using kubeadm

How to reproduce it (as minimally and precisely as possible):
apt-get install -y apt-transport-https
sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
sudo echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee
/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
apt-get update

apt-get install -y kubelet kubeadm kubectl kubernetes-cni

swapoff -a
kubeadm init

Anything else we need to know?:
This was the initial error after kubeadm init. It got stuck for a while at "[init] This might take a minute or longer if the control plane images have to be pulled."

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- There is no internet connection, so the kubelet cannot pull the following control plane images:
- gcr.io/google_containers/kube-apiserver-amd64:v1.9.3
- gcr.io/google_containers/kube-controller-manager-amd64:v1.9.3
- gcr.io/google_containers/kube-scheduler-amd64:v1.9.3

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration:
    Local Server (Ubuntu 16.04.3)

  • Kernel (e.g. uname -a):
    Linux server1 4.13.0-36-generic Link fix to cloud consoleย #40~16.04.1-Ubuntu SMP Fri Feb 16 23:25:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:
    kubeadm

  • Others:

Metadata

Assignees

No one assigned

    Labels

    sig/cluster-lifecycleCategorizes an issue or PR as relevant to SIG Cluster Lifecycle.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions