Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

static pod with gcr.io/google_containers/nginx:1.7.9 image did not start within 2m #24782

Closed
fejta opened this issue Apr 26, 2016 · 5 comments
Closed

static pod with gcr.io/google_containers/nginx:1.7.9 image did not start within 2m #24782

fejta opened this issue Apr 26, 2016 · 5 comments
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@fejta
Copy link
Contributor

fejta commented Apr 26, 2016 

• Failure in Spec Setup (BeforeEach) [120.004 seconds]
MirrorPod
/var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:121
  when create a mirror pod 
  /var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:120
    should be updated when static pod updated [BeforeEach]
    /var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:79

    Timed out after 120.000s.
    Expected
        <*errors.errorString | 0xc820387830>: {
            s: "expected the mirror pod \"static-pod-7b242f93-0b79-11e6-be53-42010af00019-tmp-node-e2e-a0bb2bf7-e2e-node-containervm-20160217-image\" to be running, got \"Pending\"",
        }
    to be nil

    /var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:56

https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/24131/node-pull-build-e2e-test/2995/build-log.txt
@fejta fejta added sig/node Categorizes an issue or PR as relevant to SIG Node. kind/flake Categorizes issue or PR as related to a flaky test. labels Apr 26, 2016
@fejta fejta mentioned this issue Apr 26, 2016
Merged
@bprashanth
Copy link
Contributor

The pod in question is static-pod-7b242f93-0b79-11e6-be53-42010af00019-tmp-node-e2e-a0bb2bf7-e2e-node-containervm-20160217-image, it was created at 6:38 and failed at 6:40 because it couldn't open the DNS config file.

@kubernetes/goog-node maybe a race between creating mount points and starting containers that forces startup to take > 120s?

I0426 06:40:52.657362    2700 provider.go:115] Refreshing cache for provider: *gcp_credentials.dockerConfigKeyProvider
E0426 06:40:52.657607    2700 kubelet.go:1364] hostname for pod:"static-pod-7b242f93-0b79-11e6-be53-42010af00019-tmp-node-e2e-a0bb2bf7-e2e-node-containervm-20160217-image" was longer than 63. Truncated hostname to :"static-pod-7b242f93-0b79-11e6-be53-42010af00019-tmp-node-e2e-a0"
E0426 06:40:52.657679    2700 manager.go:1957] Error running pod "static-pod-7b242f93-0b79-11e6-be53-42010af00019-tmp-node-e2e-a0bb2bf7-e2e-node-containervm-20160217-image_mirror-pod(43524d206ecf8183262e88b505a7e96d)" container "test": GenerateRunContainerOptions: open /var/lib/kubelet/pods/43524d206ecf8183262e88b505a7e96d/etc-hosts: no such file or directory
E0426 06:40:52.657721    2700 pod_workers.go:138] Error syncing pod 43524d206ecf8183262e88b505a7e96d, skipping: failed to "StartContainer" for "test" with RunContainerError: "GenerateRunContainerOptions: open /var/lib/kubelet/pods/43524d206ecf8183262e88b505a7e96d/etc-hosts: no such file or directory"

@Random-Liu
Copy link
Member

Random-Liu commented May 3, 2016 •
edited
Loading

@bprashanth Looks like SyncPod was stuck before creating mount points (even before GenerateRunContainerOptions), because we don't do any mount before:

E0426 06:40:52.657607    2700 kubelet.go:1364] hostname for pod:"static-pod-7b242f93-0b79-11e6-be53-42010af00019-tmp-node-e2e-a0bb2bf7-e2e-node-containervm-20160217-image" was longer than 63. Truncated hostname to :"static-pod-7b242f93-0b79-11e6-be53-42010af00019-tmp-node-e2e-a0"`.

I suspect it was stuck by image pulling, because the mirror pod test is the only one using "nginx:1.7.9", while the failure happened when the first test case run.

@0xmichalis 0xmichalis mentioned this issue May 9, 2016
Merged
This was referenced May 13, 2016
Closed
Closed
@fgrzadkowski
Copy link
Contributor

Happened again:

02:03:33 STEP: create the static pod
02:03:33 STEP: wait for the mirror pod to be running
02:03:33 STEP: delete the static pod
02:03:33 STEP: wait for the mirror pod to disappear
02:03:33 MirrorPod
02:03:33 /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:121
02:03:33   when create a mirror pod 
02:03:33   /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:120
02:03:33     should be updated when static pod updated [BeforeEach]
02:03:33     /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:79
02:03:33 
02:03:33     Timed out after 120.000s.
02:03:33     Expected
02:03:33         <*errors.errorString | 0xc82000ebb0>: {
02:03:33             s: "expected the mirror pod \"static-pod-886881ff-1e66-11e6-aecc-42010af00026-tmp-node-e2e-b4458175-e2e-node-coreos-stable20160218-image\" to be running, got \"Pending\"",
02:03:33         }
02:03:33     to be nil
02:03:33 
02:03:33     /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:56

@fgrzadkowski fgrzadkowski added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label May 20, 2016
@yujuhong
Copy link
Contributor

@fgrzadkowski could you post a link to the build and/or logs?

@Random-Liu
Copy link
Member

I still suspect it is caused by a slow image pulling. #25944 would fix this.

@fejta fejta closed this as completed May 23, 2016
@stannnous stannnous mentioned this issue Mar 21, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fejta @Random-Liu @yujuhong @fgrzadkowski @bprashanth