This seems like #25277, I saw this in log:

May 23 05:38:36.201: INFO: At {2016-05-23 05:27:49 -0700 PDT} - event for node-problem-detector-v0.1-5tz3a: {kubelet e2e-scalability-minion-pwnb} FailedSync: Error syncing pod, skipping: failed to "StartContainer" for "node-problem-detector" with ErrImagePull: "net/http: request canceled"

Yes it is dupe of #25277

I am closing this issue, and we should fix the image pulling flakiness separately. Hope proposal #25784 can get us to a better situation.

I don't think #25784 will solve problems like this, where image pulling failed at the cluster bringup stage. The node in question tried pulling the image 5 times and all failed.

OK. I think this is caused by the docker operation timeout that was added to our docker client.

When setting the default context timeout to "12s", the request timed out after 12s exactly. The error "net/http: request canceled" matches what we've seen in the tests since the timeout was added.

$ time ./image_puller --image=busybox
Connecting to docker on unix:///var/run/docker.sock...
Pulling image "busybox"...
First phase succeeded
Failed to pull image: net/http: request canceled

real    0m12.080s
user    0m0.014s
sys 0m0.080s

At cluster bringup, the registry can be slow due to all the pulling requests from the nodes, which in turn causes the 2 minute timeout. This also prevents kubelet from pulling any image that needs > 2min (default timeout) to pull. I'll send a PR to disable timeout for image pulling for now.

p.s.1: @Random-Liu, the timeout applies to the entire image pulling operation in my tests, as opposed to the initial response you mentioned last time.
p.s.2: The simple image pulling code used for reproducing this issue: https://gist.github.com/yujuhong/fafeb49c7b69d40573feeabdc7789142

@yujuhong I checked the code of engine-api again, you are right, the timeout is applied on the whole image pulling:
https://github.com/docker/engine-api/blob/master/client/transport/cancellable/cancellable.go#L77-L84

I didn't notice that the client hold the context itself in another goroutine, and checks if the context is Done(). Whenever the context is done, it will cancel the request, that's why we get the error "net/http: request canceled".

For normal case, we should get the timeout from the Context. But for image pulling, we didn't check the context, then the "net/http: request canceled" error surfaces.

For normal case, we should get the timeout from the Context. But for image pulling, we didn't check the context, then the "net/http: request canceled" error surfaces.

Yes, I also noticed that :-)

@yujuhong Thanks a lot for finding this. Sorry I didn't catch this before. :(
If so, maybe we should make the image pull timeout configurable or update image pulling progress to surface the original image pulling stuck issue.

@yujuhong Thanks a lot for finding this. Sorry I didn't catch this before. :(

No worries. In only checked because the test failed to pull the same image for 5 times with the "request canceled" message :)

If so, maybe we should make the image pull timeout configurable or update image pulling progress to surface the original image pulling stuck issue.

Checking image progress sounds reasonable to me. Maybe we can detect and cancel the request when the progress is stalled for a prolonged period of time.

@yujuhong I'll send a PR to at least properly log the image pulling progress.

SGTM. Thanks!