Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Image pulling failing with bearer token errors #27451

Closed
bprashanth opened this issue Jun 15, 2016 · 16 comments
Closed

Image pulling failing with bearer token errors #27451

bprashanth opened this issue Jun 15, 2016 · 16 comments
Assignees
@dchen1107
Labels
area/images-registry area/platform/gce kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@bprashanth
Copy link
Contributor

Seeing tests fail because image pulling failed, because of:

17:24:09 pulling gcr.io/google_containers/pause-amd64:3.0
Error response from daemon: name invalid: Requested repository does not match bearer token resource: google_containers/pause-amd64
17:24:09 pulling gcr.io/google_containers/porter:cd5cb5791ebaa8641955f0e8c2a9bed669b1eaab
Error response from daemon: name invalid: Requested repository does not match bearer token resource: google_containers/porter
17:24:09 pulling gcr.io/google_containers/portforwardtester:1.0

#26134 (comment)
http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/44825/consoleFull

@kubernetes/goog-node did we recently change image pulling?
@bprashanth bprashanth added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Jun 15, 2016
@bprashanth bprashanth mentioned this issue Jun 15, 2016
Merged
@yujuhong
Copy link
Contributor

Not that I know of.

@yifan-gu
Copy link
Contributor

Seeing this error as well in the e2e tests result #27220

@yifan-gu yifan-gu mentioned this issue Jun 15, 2016
Merged
@ikehz ikehz mentioned this issue Jun 15, 2016
Merged
@bprashanth
Copy link
Contributor Author

in #27415 (comment) this failed on node e2e-gce-agent-pr-28-0-minion-group-7mx6 but passed on some other nodes like e2e-gce-agent-pr-28-0-minion-group-a8gv.
@k8s-oncall fyi

@ikehz
Copy link
Contributor

ikehz commented Jun 15, 2016

I'm seeing this error also pulling images when trying to build: #27453 (comment)

@ikehz
Copy link
Contributor

ikehz commented Jun 15, 2016

cc @roberthbailey

@bprashanth bprashanth added kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jun 15, 2016
@bprashanth bprashanth mentioned this issue Jun 15, 2016
Merged
@bprashanth
Copy link
Contributor Author

Yeah this is foreboding but I don't have the time to dig, maybe we shoudl reach out to registry people to check? @kubernetes/goog-testing

@ikehz ikehz self-assigned this Jun 15, 2016
@ikehz
Copy link
Contributor

ikehz commented Jun 15, 2016

I can have a look.

@Random-Liu Random-Liu mentioned this issue Jun 15, 2016
Closed
@girishkalele girishkalele changed the title Image pulling faliling with bearer token errors Image pulling failing with bearer token errors Jun 15, 2016
@mml
Copy link
Contributor

mml commented Jun 15, 2016

@quinton-hoole mentioned this as well.

@mml
Copy link
Contributor

mml commented Jun 15, 2016

And I saw this just now. https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/27390/node-pull-build-e2e-test/10692/build-log.txt

@mml mml added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jun 15, 2016
@mml mml mentioned this issue Jun 15, 2016
Merged
@ixdy ixdy mentioned this issue Jun 15, 2016
Merged
@mml mml mentioned this issue Jun 15, 2016
Merged
@wojtek-t
Copy link
Member

And I saw this error 4 times in gke-large suite yesterday:
http://kubekins.dls.corp.google.com:8080/view/Scalability/job/kubernetes-e2e-gke-large-cluster/49/consoleText

[4 different tests failed exactly because of that reason - see #27527 ]

@wojtek-t wojtek-t mentioned this issue Jun 16, 2016
Closed
4 tasks
@gmarek gmarek mentioned this issue Jun 16, 2016
Closed
This was referenced Jun 16, 2016
Merged
Merged
@wojtek-t
Copy link
Member

I think at least understanding this one should be in 1.3 milestone. @dchen1107 ?

@yujuhong yujuhong added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. labels Jun 16, 2016
@yujuhong
Copy link
Contributor

Based on the internal email thread, the GCR team was able to reproduce this failure with docker pull and gcloud docker pull.
They have also started rolling back their release to see if this would be fixed.

@ikehz
Copy link
Contributor

ikehz commented Jun 16, 2016

The GCR team rolled back around 4 p.m. yesterday, and it doesn't look like we've seen any issues since then.

@ikehz ikehz removed their assignment Jun 16, 2016
@yujuhong
Copy link
Contributor

Thanks @ihmccreery. I am closing this issue since we haven't seen any failure since the rollback. Will reopen if there are any new failure.

@ikehz
Copy link
Contributor

ikehz commented Jun 16, 2016

Yes, the root cause on the GCR side sounds like it's understood, so we're good.

@wojtek-t
Copy link
Member

Yeah - the failures I linked were also from before 4pm

@spxtr spxtr mentioned this issue Jun 21, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dchen1107 dchen1107

Labels
area/images-registry area/platform/gce kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@mml @ikehz @yifan-gu @Random-Liu @dchen1107 @yujuhong @wojtek-t @bprashanth