Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix podstatus issue caused by docker's resource temporarily unavailable issue #4376

Merged
merged 2 commits into from
Feb 12, 2015
Merged

Fix podstatus issue caused by docker's resource temporarily unavailable issue #4376

merged 2 commits into from
Feb 12, 2015
+20 −12

Conversation

dchen1107
Copy link
Member

Fix #4188. I tested this PR by pushing it to a running cluster with heavy load (300 pods on 4 nodes)

$ ./cluster/kubectl.sh get pods -l app=hostnames | awk '{print $(NF-2)}' | sort | uniq -c
Project: golden-system-455
Zone: us-central1-b
Running: ./cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames
1 HOST
110 kubernetes-minion-1.c.golden-system-455.internal/130.211.172.228
113 kubernetes-minion-2.c.golden-system-455.internal/104.154.62.162
62 kubernetes-minion-3.c.golden-system-455.internal/104.154.62.20
15 kubernetes-minion-4.c.golden-system-455.internal/104.154.55.124

And periodically checking podstatus of all pods for a couple of hours, and there is no flaps among Running, Pending and Unknown anymore:

$ while true; do ./cluster/kubectl.sh get pods -l app=hostnames | grep -v Running; date ; sleep 10; done 
Project: golden-system-455
Zone: us-central1-b
Running: ./cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames
POD                 IP                  CONTAINER(S)        IMAGE(S)                    HOST                                                               LABELS              STATUS
Wed Feb 11 21:57:10 PST 2015
...
Running: ./cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames
POD                 IP                  CONTAINER(S)        IMAGE(S)                    HOST                                                               LABELS              STATUS
Wed Feb 11 23:58:38 PST 2015
Project: golden-system-455
Zone: us-central1-b
Running: ./cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames
POD                 IP                  CONTAINER(S)        IMAGE(S)                    HOST                                                               LABELS              STATUS
Wed Feb 11 23:58:55 PST 2015

@erictune
Copy link
Member

LGTM

erictune added a commit that referenced this pull request Feb 12, 2015
@erictune
Merge pull request #4376 from dchen1107/podstatus
dd8f335 
Fix podstatus issue caused by docker's resource temporarily unavailable issue
@erictune erictune merged commit dd8f335 into kubernetes:master Feb 12, 2015
This was referenced Feb 12, 2015
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@davidopp davidopp

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

API status for pods is flaky
3 participants
@dchen1107 @erictune @davidopp