Do not query the metadata server to find out if running on GCE. Retry metadata server query for gcr if running on gce. #28871

vishh · 2016-07-13T00:42:52Z

Retry the logic for determining is gcr is enabled to workaround metadata unavailability.

Note: This patch does not retry fetching registry credentials.

vishh · 2016-07-13T00:43:51Z

cc @Random-Liu

vishh · 2016-07-13T00:50:56Z

cc @zmerlynn @roberthbailey

liggitt · 2016-07-13T01:01:25Z

Was this related to the PR @derekwaynecarr had opened?

yujuhong · 2016-07-13T01:04:23Z

pkg/credentialprovider/gcp/metadata.go

+		value, err := credentialprovider.ReadUrl(metadataScopes+"?alt=json", g.Client, metadataHeader)
+		if err == nil {
+			break
+		}


Should this be a hot loop? Backing off or sleeping seems better.

vishh · 2016-07-13T01:09:05Z

@liggitt #28539 seems orthogonal. WDYT?

liggitt · 2016-07-13T01:09:14Z

Does this remove the need for #28539, or is the timeout still a concern?

liggitt · 2016-07-13T01:10:16Z

Not sure, I think that was mostly to avoid hangs in non-GCE envs, so this might prevent that entirely?

yujuhong · 2016-07-13T02:06:27Z

pkg/credentialprovider/gcp/metadata.go

+	for {
+		value, err = credentialprovider.ReadUrl(metadataScopes+"?alt=json", g.Client, metadataHeader)
+		if err == nil {
+			break


Don't think my previous comment was addressed: this shouldn't run in such a hot loop.

Wouldn't ReadUrl have an in-built timeout? I don't think this logic will busy wait. I can add a delay if it will be useful.

Even if there is a timeout, trying immediately may not be good. If the metadata server is overwhelmed by requests already, adding a delay will help.

yujuhong · 2016-07-13T17:48:35Z

Failed on the containervm image in the node e2e tests.

I0713 01:15:32.908763    2975 plugins.go:56] Registering credential provider: google-dockercfg-url
I0713 01:15:32.910406    2975 config.go:156] body of failing http response: &{0xc8207d6ac0 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.911834    2975 config.go:156] body of failing http response: &{0xc8207d6cc0 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.913305    2975 config.go:156] body of failing http response: &{0xc8207d6e80 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.914465    2975 config.go:156] body of failing http response: &{0xc8207d7040 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.915502    2975 config.go:156] body of failing http response: &{0xc8207d7200 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.916609    2975 config.go:156] body of failing http response: &{0xc8207d73c0 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.917703    2975 config.go:156] body of failing http response: &{0xc8207d7580 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.918938    2975 config.go:156] body of failing http response: &{0xc8207d7740 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.919959    2975 config.go:156] body of failing http response: &{0xc8207d7900 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.920924    2975 config.go:156] body of failing http response: &{0xc8207d7b40 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.921986    2975 config.go:156] body of failing http response: &{0xc8207d7d00 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.923190    2975 config.go:156] body of failing http response: &{0xc8207d7ec0 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.924502    2975 config.go:156] body of failing http response: &{0xc820790080 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.926943    2975 config.go:156] body of failing http response: &{0xc820790240 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.927894    2975 config.go:156] body of failing http response: &{0xc820790400 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.928942    2975 config.go:156] body of failing http response: &{0xc8207905c0 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.929990    2975 config.go:156] body of failing http response: &{0xc820790780 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.931136    2975 config.go:156] body of failing http response: &{0xc820790940 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.932207    2975 config.go:156] body of failing http response: &{0xc820790b00 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.933227    2975 config.go:156] body of failing http response: &{0xc820790cc0 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.934656    2975 config.go:156] body of failing http response: &{0xc820790e80 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.935695    2975 config.go:156] body of failing http response: &{0xc820791040 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.936742    2975 config.go:156] body of failing http response: &{0xc820791200 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.937971    2975 config.go:156] body of failing http response: &{0xc820791400 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.939060    2975 config.go:156] body of failing http response: &{0xc8207915c0 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.940642    2975 config.go:156] body of failing http response: &{0xc820791780 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.941890    2975 config.go:156] body of failing http response: &{0xc820791940 {0 0} false <nil> 0x67da70 0x67da10}
I0713 01:15:32.943356    2975 config.go:156] body of failing http response: &{0xc820504600 {0 0} false <nil> 0x67da70 0x67da10}

I didn't dig into failure, but FWIW, I checked and container registry registration always fails even in a green node e2e run.

roberthbailey · 2016-07-13T18:23:45Z

can you squash commits?

zmerlynn · 2016-07-13T18:28:45Z

This looks fine, but I'm heading out soon. Squash and I'll defer to @roberthbailey for final.

Random-Liu · 2016-07-13T21:41:45Z

E0713 17:57:34.501324    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:34.503743    2935 config.go:156] body of failing http response: &{0xc8207a4500 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:34.503769    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:34.507202    2935 config.go:156] body of failing http response: &{0xc8207a4700 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:34.507238    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:34.512487    2935 config.go:156] body of failing http response: &{0xc8207a4900 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:34.512522    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:34.522237    2935 config.go:156] body of failing http response: &{0xc8207a4b00 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:34.522292    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:34.540172    2935 config.go:156] body of failing http response: &{0xc8207a4d40 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:34.540241    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:34.574401    2935 config.go:156] body of failing http response: &{0xc8207a4f40 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:34.574456    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:34.640219    2935 config.go:156] body of failing http response: &{0xc8207a5140 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:34.640286    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:34.770057    2935 config.go:156] body of failing http response: &{0xc8207a59c0 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:34.770135    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:35.028187    2935 config.go:156] body of failing http response: &{0xc8207a5d80 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:35.028258    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:35.542159    2935 config.go:156] body of failing http response: &{0xc8207a5f80 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:35.542240    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:36.568324    2935 config.go:156] body of failing http response: &{0xc8204fc2c0 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:36.568400    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:38.618818    2935 config.go:156] body of failing http response: &{0xc8204fca80 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:38.618909    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:42.717029    2935 config.go:156] body of failing http response: &{0xc8204fce80 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:42.717093    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:44.420624    2935 config.go:280] Setting pods for source file
I0713 17:57:50.912456    2935 config.go:156] body of failing http response: &{0xc8204fd280 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:57:50.912533    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json
I0713 17:57:54.420888    2935 config.go:280] Setting pods for source file
I0713 17:58:04.421878    2935 config.go:280] Setting pods for source file
I0713 17:58:07.300525    2935 config.go:156] body of failing http response: &{0xc8204fd880 {0 0} false <nil> 0x67db80 0x67db20}
E0713 17:58:07.300594    2935 metadata.go:183] failed to Get "http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json": http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/service-accounts/default/scopes?alt=json

yujuhong · 2016-07-13T21:47:03Z

I guess we don't have service account on the node e2e vm instances.

How about checking http://metadata.google.internal./computeMetadata/v1/instance/ first to see if service-account is there?

Random-Liu · 2016-07-13T21:47:05Z

I guess It looks like the gcr credential provider is never enabled in e2e test before, because the metadata server is never reachable. Here is the log from the node e2e kubelet log:

I0713 21:39:43.320881    2963 plugins.go:56] Registering credential provider: google-dockercfg
I0713 21:39:43.321450    2963 plugins.go:56] Registering credential provider: google-dockercfg-url

However, now we change to use file based check, and it always passes, then kubelet keeps failing to access metadata server after that.

roberthbailey · 2016-07-15T20:58:10Z

@yujuhong - if this looks good to you can you apply the label?

k8s-github-robot · 2016-07-15T21:11:51Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

vishh · 2016-07-15T21:21:21Z

Bumping up the priority since this is required for v1.3

k8s-bot · 2016-07-15T21:48:54Z

GCE e2e build/test passed for commit ea1a459.

k8s-github-robot · 2016-07-15T22:27:45Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

vishh · 2016-07-15T22:39:59Z

@k8s-bot test this github issue #IGNORE

k8s-bot · 2016-07-15T22:43:12Z

GCE e2e build/test failed for commit ea1a459.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

vishh · 2016-07-15T23:00:23Z

@k8s-bot test this github issue #IGNORE

k8s-bot · 2016-07-15T23:03:03Z

GCE e2e build/test failed for commit ea1a459.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

k8s-bot · 2016-07-15T23:08:44Z

GCE e2e build/test passed for commit ea1a459.

vishh · 2016-07-18T19:03:23Z

@k8s-bot test this github issue #IGNORE

k8s-bot · 2016-07-18T19:38:13Z

GCE e2e build/test passed for commit ea1a459.

k8s-github-robot · 2016-07-18T20:54:49Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2016-07-18T21:31:48Z

GCE e2e build/test passed for commit ea1a459.

k8s-github-robot · 2016-07-18T21:32:04Z

Automatic merge from submit-queue

…upstream-release-1.3 Automatic merge from submit-queue Automated cherry pick of #28871 Cherry pick of #28871 on release-1.3.

k8s-cherrypick-bot · 2016-07-19T21:43:52Z

Commit found in the "release-1.3" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

…of-#28871-upstream-release-1.3 Automatic merge from submit-queue Automated cherry pick of kubernetes#28871 Cherry pick of kubernetes#28871 on release-1.3.

googlebot added the cla: yes label Jul 13, 2016

k8s-github-robot assigned liggitt Jul 13, 2016

k8s-github-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. release-note-label-needed labels Jul 13, 2016

vishh assigned roberthbailey and zmerlynn and unassigned liggitt Jul 13, 2016

yujuhong reviewed Jul 13, 2016
View reviewed changes

vishh force-pushed the gce-cp branch from f3de778 to b175e08 Compare July 13, 2016 01:08

Random-Liu added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed labels Jul 13, 2016

Random-Liu added this to the v1.3 milestone Jul 13, 2016

Random-Liu added the cherrypick-candidate label Jul 13, 2016

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 13, 2016

yujuhong reviewed Jul 13, 2016
View reviewed changes

vishh force-pushed the gce-cp branch from ff322e4 to 0bd7899 Compare July 13, 2016 22:00

roberthbailey assigned yujuhong Jul 15, 2016

yujuhong added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 15, 2016

vishh added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jul 15, 2016

k8s-github-robot merged commit 8eb0cf5 into kubernetes:master Jul 18, 2016

zmerlynn added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jul 18, 2016

vishh mentioned this pull request Jul 18, 2016

Automated cherry pick of #28871 #29152

Merged

vishh added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jul 18, 2016

k8s-github-robot pushed a commit that referenced this pull request Jul 19, 2016

Merge pull request #29152 from vishh/automated-cherry-pick-of-#28871-…

a6c9dbd

…upstream-release-1.3 Automatic merge from submit-queue Automated cherry pick of #28871 Cherry pick of #28871 on release-1.3.

k8s-cherrypick-bot removed the cherrypick-candidate label Jul 19, 2016

derekwaynecarr mentioned this pull request Jul 20, 2016

Fix httpclient setup for gcp credential provider to have timeout #28539

Merged

ncdc mentioned this pull request Jul 22, 2016

1.3 upstream picks openshift/origin#9790

Closed

roberthbailey mentioned this pull request Jul 24, 2016

Remove compute-rw scope from GCE nodes #8074

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not query the metadata server to find out if running on GCE. Retry metadata server query for gcr if running on gce. #28871

Do not query the metadata server to find out if running on GCE. Retry metadata server query for gcr if running on gce. #28871

vishh commented Jul 13, 2016 •

edited

Loading

vishh commented Jul 13, 2016

vishh commented Jul 13, 2016

liggitt commented Jul 13, 2016

yujuhong Jul 13, 2016

vishh commented Jul 13, 2016

liggitt commented Jul 13, 2016

liggitt commented Jul 13, 2016

yujuhong Jul 13, 2016

vishh Jul 13, 2016

yujuhong Jul 13, 2016

yujuhong commented Jul 13, 2016

roberthbailey commented Jul 13, 2016

zmerlynn commented Jul 13, 2016

Random-Liu commented Jul 13, 2016

yujuhong commented Jul 13, 2016 •

edited

Loading

Random-Liu commented Jul 13, 2016 •

edited

Loading

roberthbailey commented Jul 15, 2016

k8s-github-robot commented Jul 15, 2016

vishh commented Jul 15, 2016

k8s-bot commented Jul 15, 2016

k8s-github-robot commented Jul 15, 2016

vishh commented Jul 15, 2016

k8s-bot commented Jul 15, 2016

vishh commented Jul 15, 2016

k8s-bot commented Jul 15, 2016

k8s-bot commented Jul 15, 2016

vishh commented Jul 18, 2016

k8s-bot commented Jul 18, 2016

k8s-github-robot commented Jul 18, 2016

k8s-bot commented Jul 18, 2016

k8s-github-robot commented Jul 18, 2016

k8s-cherrypick-bot commented Jul 19, 2016

Do not query the metadata server to find out if running on GCE. Retry metadata server query for gcr if running on gce. #28871

Do not query the metadata server to find out if running on GCE. Retry metadata server query for gcr if running on gce. #28871

Conversation

vishh commented Jul 13, 2016 • edited Loading

vishh commented Jul 13, 2016

vishh commented Jul 13, 2016

liggitt commented Jul 13, 2016

yujuhong Jul 13, 2016

Choose a reason for hiding this comment

vishh commented Jul 13, 2016

liggitt commented Jul 13, 2016

liggitt commented Jul 13, 2016

yujuhong Jul 13, 2016

Choose a reason for hiding this comment

vishh Jul 13, 2016

Choose a reason for hiding this comment

yujuhong Jul 13, 2016

Choose a reason for hiding this comment

yujuhong commented Jul 13, 2016

roberthbailey commented Jul 13, 2016

zmerlynn commented Jul 13, 2016

Random-Liu commented Jul 13, 2016

yujuhong commented Jul 13, 2016 • edited Loading

Random-Liu commented Jul 13, 2016 • edited Loading

roberthbailey commented Jul 15, 2016

k8s-github-robot commented Jul 15, 2016

vishh commented Jul 15, 2016

k8s-bot commented Jul 15, 2016

k8s-github-robot commented Jul 15, 2016

vishh commented Jul 15, 2016

k8s-bot commented Jul 15, 2016

vishh commented Jul 15, 2016

k8s-bot commented Jul 15, 2016

k8s-bot commented Jul 15, 2016

vishh commented Jul 18, 2016

k8s-bot commented Jul 18, 2016

k8s-github-robot commented Jul 18, 2016

k8s-bot commented Jul 18, 2016

k8s-github-robot commented Jul 18, 2016

k8s-cherrypick-bot commented Jul 19, 2016

vishh commented Jul 13, 2016 •

edited

Loading

yujuhong commented Jul 13, 2016 •

edited

Loading

Random-Liu commented Jul 13, 2016 •

edited

Loading