Kubelet: Fail kubelet if cadvisor is not started. #29581
Merged
+3
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Random-Liu
added
kind/bug
Random-Liu
added
the
priority/important-soon
Random-Liu
force-pushed
the
panic-if-cadvisor-not-started
branch
from
18043ad to
e7908c0
Compare
k8s-github-robot
added
size/XS
Random-Liu
added
release-note-none
Random-Liu
changed the title
Random-Liu
force-pushed
the
panic-if-cadvisor-not-started
branch
from
e7908c0 to
973f2fc
Compare
|
So you are crashing Kubelet instead of failing /healthz? |
|
LGTM |
dchen1107
added
the
lgtm
|
GCE e2e build/test passed for commit 973f2fc. |
|
Automatic merge from submit-queue |
fabioy
added a commit
that referenced
this pull request
Jul 26, 2016
…9581-upstream-release-1.3 Automated cherry pick of #29581
|
Commit found in the "release-1.3" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked. |
shyamjvs
pushed a commit
to shyamjvs/kubernetes
that referenced
this pull request
Dec 1, 2016
…pick-of-#29581-upstream-release-1.3 Automated cherry pick of kubernetes#29581
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #28997.
We started cadvisor in
sync.Do(), which only run once no matter cadvisor successfully starts or not.Once it fails, kubelet will be stuck in a bad state. Kubelet could never start sync loop because there is an internal error, but kubelet would never retry starting cadvisor again.
This PR just fails kubelet when cadvisor start fails, and then relies on the babysitter to restart kubelet.
In the future, we may want to add backoff logic in the babysitter to protect the system.
On the other hand, #29492 will fix cadvisor side to prevent cadvisor failing because of these kind of transient error.
Mark P1 to match the original issue.
@dchen1107 @vishh