Skip to content

Node E2E: Failed to pre-pull image #26203

Closed
@Random-Liu

Description

@Random-Liu
I0524 18:20:26.486302    2069 e2e_node_suite_test.go:67] Pre-pulling images so that they are cached for the tests.
W0524 18:20:41.679642    2069 container_list.go:56] Could not pre-pull image %s %v output:  %sgcr.io/google_containers/pause-amd64:3.0exit status 1 [69 114 114 111 114 32 114 101 115 112 111 110 115 101 32 102 114 111 109 32 100 97 101 109 111 110 58 32 71 101 116 32 104 116 116 112 115 58 47 47 103 99 114 46 105 111 47 118 50 47 103 111 111 103 108 101 95 99 111 110 116 97 105 110 101 114 115 47 112 97 117 115 101 45 97 109 100 54 52 47 109 97 110 105 102 101 115 116 115 47 51 46 48 58 32 71 101 116 32 104 116 116 112 115 58 47 47 103 99 114 46 105 111 47 118 50 47 116 111 107 101 110 63 115 99 111 112 101 61 114 101 112 111 115 105 116 111 114 121 37 51 65 103 111 111 103 108 101 95 99 111 110 116 97 105 110 101 114 115 37 50 70 112 97 117 115 101 45 97 109 100 54 52 37 51 65 112 117 108 108 38 115 101 114 118 105 99 101 61 103 99 114 46 105 111 58 32 110 101 116 47 104 116 116 112 58 32 114 101 113 117 101 115 116 32 99 97 110 99 101 108 101 100 32 119 104 105 108 101 32 119 97 105 116 105 110 103 32 102 111 114 32 99 111 110 110 101 99 116 105 111 110 32 40 67 108 105 101 110 116 46 84 105 109 101 111 117 116 32 101 120 99 101 101 100 101 100 32 119 104 105 108 101 32 97 119 97 105 116 105 110 103 32 104 101 97 100 101 114 115 41 10]

@ncdc translated the string part:

Error response from daemon: Get https://gcr.io/v2/google_containers/pause-amd64/manifests/3.0: Get https://gcr.io/v2/token?scope=repository%3Agoogle_containers%2Fpause-amd64%3Apull&service=gcr.io: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Since the node e2e test uses "docker pull" command directly, so there may be a docker cli operation timeout.

Activity

added
sig/nodeCategorizes an issue or PR as relevant to SIG Node.
kind/flakeCategorizes issue or PR as related to a flaky test.
on May 24, 2016
dchen1107

dchen1107 commented on May 24, 2016

@dchen1107
Member

Is this part of runtime conformance tests?

Random-Liu

Random-Liu commented on May 24, 2016

@Random-Liu
MemberAuthor

@dchen1107 No. This happens during prepulling all node e2e images. Ref #25944.

dchen1107

dchen1107 commented on May 24, 2016

@dchen1107
Member

Ok, since it is a best-effort thing to help with deflake the test failures caused by slow pulling, can we either add retry logic or at least not fail the entire test suite?

cc/ @pwittrock

pwittrock

pwittrock commented on May 24, 2016

@pwittrock
Member

Retrying pre-pulling failed images SGTM. Maybe 5 times each?

pwittrock

pwittrock commented on May 24, 2016

@pwittrock
Member

It is nice that the test failure reason is obvious :)

janetkuo

janetkuo commented on May 25, 2016

@janetkuo
Member

dial tcp 74.125.202.82:443: i/o timeout when pulling images:

https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/26312/node-pull-build-e2e-test/7927/build-log.txt

I0525 22:18:41.539120     941 e2e_node_suite_test.go:81] Pre-pulling images so that they are cached for the tests.
W0525 22:20:48.249600     941 container_list.go:56] Could not pre-pull image gcr.io/google_containers/netexec:1.4 exit status 1 output:  Pulling repository gcr.io/google_containers/netexec
76fe3da0683d: Pulling image (1.4) from gcr.io/google_containers/netexec
76fe3da0683d: Pulling image (1.4) from gcr.io/google_containers/netexec, endpoint: https://gcr.io/v1/
76fe3da0683d: Error pulling image (1.4) from gcr.io/google_containers/netexec, endpoint: https://gcr.io/v1/, Get https://gcr.io/v1/images/76fe3da0683d17e9699f5ff66e1b7ce488a9c8abcad6c412547e75915beb12d9/ancestry: dial tcp 74.125.202.82:443: i/o timeout
76fe3da0683d: Error pulling image (1.4) from gcr.io/google_containers/netexec, Get https://gcr.io/v1/images/76fe3da0683d17e9699f5ff66e1b7ce488a9c8abcad6c412547e75915beb12d9/ancestry: dial tcp 74.125.202.82:443: i/o timeout
Error pulling image (1.4) from gcr.io/google_containers/netexec, Get https://gcr.io/v1/images/76fe3da0683d17e9699f5ff66e1b7ce488a9c8abcad6c412547e75915beb12d9/ancestry: dial tcp 74.125.202.82:443: i/o timeout
Failure [126.712 seconds]
[BeforeSuite] BeforeSuite 
/var/lib/jenkins/jobs/node-pull-build-e2e-test/workspace@2/go/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:100

  Expected error:
      <*exec.ExitError | 0xc820443f80>: {
          ProcessState: {
              pid: 1024,
              status: 256,
              rusage: {
                  Utime: {Sec: 0, Usec: 14000},
                  Stime: {Sec: 0, Usec: 2000},
                  Maxrss: 20016,
                  Ixrss: 0,
                  Idrss: 0,
                  Isrss: 0,
                  Minflt: 829,
                  Majflt: 0,
                  Nswap: 0,
                  Inblock: 0,
                  Oublock: 0,
                  Msgsnd: 0,
                  Msgrcv: 0,
                  Nsignals: 0,
                  Nvcsw: 823,
                  Nivcsw: 136,
              },
          },
          Stderr: nil,
      }
      exit status 1
  not to have occurred

  /var/lib/jenkins/jobs/node-pull-build-e2e-test/workspace@2/go/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:83
------------------------------
I0525 22:20:48.252068     941 e2e_node_suite_test.go:108] Tests Finished

Ran 16 of 16 Specs in 126.713 seconds
FAIL! -- 0 Passed | 16 Failed | 0 Pending | 0 Skipped --- FAIL: TestE2eNode (126.71s)
FAIL
added a commit that references this issue on May 26, 2016

Merge pull request #26321 from vishh/retry-pre-pull

5 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/dockerkind/flakeCategorizes issue or PR as related to a flaky test.sig/nodeCategorizes an issue or PR as relevant to SIG Node.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Node E2E: Failed to pre-pull image · Issue #26203 · kubernetes/kubernetes