Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

node_e2e flake: kubelet_test, image_conformance_test, e2e_node_suite_test #26004

Closed
wonderfly opened this issue May 21, 2016 · 7 comments
Closed

node_e2e flake: kubelet_test, image_conformance_test, e2e_node_suite_test #26004

wonderfly opened this issue May 21, 2016 · 7 comments
Labels
area/test kind/flake Categorizes issue or PR as related to a flaky test. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@wonderfly
Copy link
Contributor

https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/25701/node-pull-build-e2e-test/7060/build-log.txt

------------------------------
• Failure [20.010 seconds]
Container runtime Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:314
  container runtime conformance blackbox test
  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:313
    when running a container with invalid image
    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:306
      it should run with the expected status [Conformance] [It]
      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:305

      Expected error:
          <*errors.errorString | 0xc82039fb20>: {
              s: "Failed to get container status",
          }
          Failed to get container status
      not to have occurred

      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:287
------------------------------
STEP: Creating a host exec pod
• Failure [120.071 seconds]
PrivilegedPod
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:83
  should test privileged pod [It]
  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:82

  Failed waiting for pod "privileged-pods/hostexec" to become running: Time out while waiting for pod privileged-pods/hostexec to become running; current status: {Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:<nil> InitContainerStatuses:[] ContainerStatuses:[]}
  Expected error:
      <*errors.errorString | 0xc8204dd330>: {
          s: "Time out while waiting for pod privileged-pods/hostexec to become running; current status: {Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:<nil> InitContainerStatuses:[] ContainerStatuses:[]}",
      }
      Time out while waiting for pod privileged-pods/hostexec to become running; current status: {Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:<nil> InitContainerStatuses:[] ContainerStatuses:[]}
  not to have occurred

  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:210
------------------------------
W0520 22:12:02.102156     932 request.go:347] Field selector: v1 - pods - metadata.name - busybox-scheduling-b70d3785-1ed6-11e6-bc67-42010af00031: need to check if this is versioned correctly.
STEP: Building a namespace api object
May 20 22:12:02.088: INFO: Skipping waiting for service account
May 20 22:17:02.103: INFO: Unexpected error occurred: timed out waiting for the condition
STEP: Collecting resource usage data
May 20 22:17:02.111: INFO: Closed stop channel. Waiting for 1 workers
May 20 22:17:02.111: INFO: Closing worker for tmp-node-e2e-9a70fd3b-e2e-node-coreos-stable20160218-image
May 20 22:17:02.111: INFO: Waitgroup finished.
May 20 22:17:02.111: INFO: Unknown output type: . Skipping.
May 20 22:17:02.111: INFO: Waiting up to 1m0s for all nodes to be ready
May 20 22:17:02.114: INFO: Found DeleteNamespace=false, skipping namespace deletion!
• Failure [300.029 seconds]
[k8s.io] Kubelet
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:508
  when scheduling a busybox command in a pod
  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:81
    it should print the output to logs [It]
    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:80

    Expected error:
        <*errors.errorString | 0xc82000fea0>: {
            s: "timed out waiting for the condition",
        }
        timed out waiting for the condition
    not to have occurred

    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:68
------------------------------
STEP: Building a namespace api object
May 20 22:17:02.117: INFO: Skipping waiting for service account
STEP: Collecting resource usage data
May 20 22:18:02.128: INFO: Closed stop channel. Waiting for 1 workers
May 20 22:18:02.128: INFO: Closing worker for tmp-node-e2e-9a70fd3b-e2e-node-coreos-stable20160218-image
May 20 22:18:02.128: INFO: Waitgroup finished.
May 20 22:18:02.128: INFO: Unknown output type: . Skipping.
May 20 22:18:02.128: INFO: Waiting up to 1m0s for all nodes to be ready
May 20 22:18:02.132: INFO: Found DeleteNamespace=false, skipping namespace deletion!
• Failure [60.018 seconds]
[k8s.io] Kubelet
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:508
  when scheduling a read only busybox container
  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:124
    it should not write to root filesystem [It]
    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:123

    Timed out after 60.000s.
    Expected
        <string>: 
    to equal
        <string>: sh: can't create /file: Read-only file system


    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:122
------------------------------
W0520 22:18:02.140060     932 request.go:347] Field selector: v1 - pods - metadata.name - stats-busybox-b77a134c-1ed8-11e6-bc67-42010af000310: need to check if this is versioned correctly.
STEP: Building a namespace api object
May 20 22:18:02.134: INFO: Skipping waiting for service account
May 20 22:23:02.140: INFO: Unexpected error occurred: timed out waiting for the condition
STEP: Collecting resource usage data
May 20 22:23:02.141: INFO: Closed stop channel. Waiting for 1 workers
May 20 22:23:02.141: INFO: Closing worker for tmp-node-e2e-9a70fd3b-e2e-node-coreos-stable20160218-image
May 20 22:23:02.141: INFO: Waitgroup finished.
May 20 22:23:02.141: INFO: Unknown output type: . Skipping.
May 20 22:23:02.141: INFO: Waiting up to 1m0s for all nodes to be ready
May 20 22:23:02.143: INFO: Found DeleteNamespace=false, skipping namespace deletion!
• Failure [300.011 seconds]
[k8s.io] Kubelet
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:508
  metrics api
  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:165
    when querying /stats/summary
    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:164
      it should report resource usage through the stats api [It]
      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:163

      Expected error:
          <*errors.errorString | 0xc82000fea0>: {
              s: "timed out waiting for the condition",
          }
          timed out waiting for the condition
      not to have occurred

      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:216
------------------------------
E0520 22:25:02.144225     932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:25:02.144281     932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:25:02.144414     932 provider.go:115] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
I0520 22:25:02.144545     932 docker.go:184] Pulling image gcr.io/google_containers/busybox:1.24 without credentials
• Failure in Spec Setup (BeforeEach) [240.002 seconds]
Image Container Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:138
  image conformance blackbox test
  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:137
    when testing images that exist
    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:84
      It should present successfully [Conformance] [BeforeEach]
      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:66

      Timed out after 120.001s.
      Expected error:
          <*errors.errorString | 0xc8202de490>: {
              s: "image pull failed for gcr.io/google_containers/busybox:1.24, this may be because there are no credentials on this request.  details: (operation timeout: context deadline exceeded)",
          }
          image pull failed for gcr.io/google_containers/busybox:1.24, this may be because there are no credentials on this request.  details: (operation timeout: context deadline exceeded)
      not to have occurred

      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:55
------------------------------
E0520 22:29:02.146143     932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:29:02.146170     932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:29:02.146236     932 docker.go:184] Pulling image gcr.io/google_containers/busybox:1.24 without credentials
• Failure in Spec Setup (BeforeEach) [240.002 seconds]
Image Container Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:138
  image conformance blackbox test
  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:137
    when testing images that exist
    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:84
      should list pulled images [Conformance] [BeforeEach]
      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:75

      Timed out after 120.000s.
      Expected error:
          <*errors.errorString | 0xc8202de4b0>: {
              s: "image pull failed for gcr.io/google_containers/busybox:1.24, this may be because there are no credentials on this request.  details: (operation timeout: context deadline exceeded)",
          }
          image pull failed for gcr.io/google_containers/busybox:1.24, this may be because there are no credentials on this request.  details: (operation timeout: context deadline exceeded)
      not to have occurred

      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:55
------------------------------
E0520 22:33:02.147760     932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:33:02.147787     932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:33:02.147874     932 provider.go:115] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
I0520 22:33:02.147973     932 docker.go:184] Pulling image foo.com/foo/fooimage:latest without credentials
E0520 22:37:02.148580     932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:37:02.148619     932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:37:02.148949     932 docker.go:184] Pulling image gcr.io/google_containers/not_exist:latest without credentials
• Failure [840.004 seconds]
Image Container Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:138
  image conformance blackbox test
  /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:137
    when testing image that does not exist
    /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:136
      it should not present images [Conformance] [It]
      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:111

      Expected error:
          <dockertools.operationTimeout>: {
              err: {
                  s: "context deadline exceeded",
              },
          }
          operation timeout: context deadline exceeded
      not to have occurred

      /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:108
------------------------------
E0520 22:47:01.562282     932 manager.go:226] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
W0520 22:47:01.562307     932 manager.go:227] Using fallback default of /var/lib/docker for location of Docker runtime
I0520 22:47:01.562377     932 provider.go:115] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
I0520 22:47:01.562486     932 docker.go:184] Pulling image foo.com/foo/fooimage:latest without credentials


Summarizing 8 Failures:

[Fail] Container runtime Conformance Test container runtime conformance blackbox test when running a container with invalid image [It] it should run with the expected status [Conformance] 
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:287

[Fail] PrivilegedPod [It] should test privileged pod 
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:210

[Fail] 
---------------------------------------------------------
Received interrupt.  Running AfterSuite...
^C again to terminate immediately
I0520 22:48:41.805703     932 e2e_node_suite_test.go:84] Tests Finished
[k8s.io] Kubelet when scheduling a busybox command in a pod [It] it should print the output to logs 
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:68

[Fail] [k8s.io] Kubelet when scheduling a read only busybox container [It] it should not write to root filesystem 
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:122

[Fail] [k8s.io] Kubelet metrics api when querying /stats/summary [It] it should report resource usage through the stats api 
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:216

[Fail] Image Container Conformance Test image conformance blackbox test when testing images that exist [BeforeEach] It should present successfully [Conformance] 
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:55

[Fail] Image Container Conformance Test image conformance blackbox test when testing images that exist [BeforeEach] should list pulled images [Conformance] 
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:55

[Fail] Image Container Conformance Test image conformance blackbox test when testing image that does not exist [It] it should not present images [Conformance] 
/var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/test/e2e_node/image_conformance_test.go:108

Ran 18 of 18 Specs in 2699.365 seconds
FAIL! -- 4 Passed | 8 Failed | 0 Pending | 0 Skipped I0520 22:48:41.809756     932 e2e_node_suite_test.go:118] Process Log For Failed Suite On tmp-node-e2e-9a70fd3b-e2e-node-coreos-stable20160218-image
-------------------------------------------------------------
@wonderfly wonderfly mentioned this issue May 21, 2016
Merged
@Random-Liu
Copy link
Member

Random-Liu commented May 21, 2016 •
edited
Loading

@wonderfly This is exactly the same issue with the last one you failed. #25993

@Random-Liu
Copy link
Member

Random-Liu commented May 21, 2016 •
edited
Loading

Looks like there is something wrong with the coreos imag:

@wonderfly
Copy link
Contributor Author

Yeah, some of them I have seen before, but not all, e.g., the kubelet_test. I didn't bother to file individual bugs, just to get k8s-bot rerun tests for my PR. Feel free to mark it as duplicated.

@Random-Liu
Copy link
Member

@wonderfly No, I mean the root cause is the same. :) Docker hung.

@Random-Liu Random-Liu mentioned this issue May 24, 2016
Merged
@Random-Liu
Copy link
Member

Another occurence: https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/26145/node-pull-build-e2e-test/7658/build-log.txt

@Random-Liu
Copy link
Member

Random-Liu commented May 24, 2016 •
edited
Loading

It's wired that each time docker hung starts during container runtime conformance test. @liangchenye
I believe it is related with the docker hung problem.

@Random-Liu Random-Liu added area/test sig/node Categorizes an issue or PR as relevant to SIG Node. kind/flake Categorizes issue or PR as related to a flaky test. labels May 24, 2016
@Random-Liu
Copy link
Member

Close this one and track the issue at #25993.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/test kind/flake Categorizes issue or PR as related to a flaky test. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Random-Liu @wonderfly