Merge pull request #29581 from Random-Liu/panic-if-cadvisor-not-started

Automatic merge from submit-queue Kubelet: Fail kubelet if cadvisor is not started. Fixes #28997. We started cadvisor in `sync.Do()`, which only run once no matter cadvisor successfully starts or not. Once it fails, kubelet will be stuck in a bad state. Kubelet could never start sync loop because there is an internal error, but kubelet would never retry starting cadvisor again. This PR just fails kubelet when cadvisor start fails, and then relies on the babysitter to restart kubelet. In the future, we may want to add backoff logic in the babysitter to protect the system. On the other hand, #29492 will fix cadvisor side to prevent cadvisor failing because of these kind of transient error. Mark P1 to match the original issue. @dchen1107 @vishh
kubernetes · Jul 26, 2016 · ed3a29b · ed3a29b
2 parents d038e22 + 973f2fc
commit ed3a29b
Showing 1 changed file with 3 additions and 1 deletion.
diff --git a/pkg/kubelet/kubelet.go b/pkg/kubelet/kubelet.go
@@ -914,7 +914,9 @@ func (kl *Kubelet) initializeModules() error {
 // initializeRuntimeDependentModules will initialize internal modules that require the container runtime to be up.
 func (kl *Kubelet) initializeRuntimeDependentModules() {
 	if err := kl.cadvisor.Start(); err != nil {
-		kl.runtimeState.setInternalError(fmt.Errorf("failed to start cAdvisor %v", err))
+		// Fail kubelet and rely on the babysitter to retry starting kubelet.
+		// TODO(random-liu): Add backoff logic in the babysitter
+		glog.Fatalf("Failed to start cAdvisor %v", err)
 	}
 	// eviction manager must start after cadvisor because it needs to know if the container runtime has a dedicated imagefs
 	if err := kl.evictionManager.Start(kl, kl.getActivePods, evictionMonitoringPeriod); err != nil {