Closed
Description
A lot of tests have started failing in kubernetes-e2e-gke since build 7657
http://kubekins.dls.corp.google.com/job/kubernetes-e2e-gke/7657/
One symptom is the missing KubeDNS service: 06:17:52 May 24 06:17:46.123: INFO: Missing KubeDNS in kubectl cluster-info
I grepped "kube-dns" in the build.log, which dumped the list of pods multiple times due to test failures, and it was nowhere to be seen.
The only change in the build is #26159
Activity
yujuhong commentedon May 24, 2016
Related failures: #26172, #26175, #26171, #26168, #26164, #26191, #26194
yujuhong commentedon May 24, 2016
/cc @k8s-oncall, this has blocked the submit queue for hours.
andyzheng0831 commentedon May 24, 2016
I find what is wrong. The PR #26159 is incorrect. Here is error log:
configure-helper.sh[1093]: /home/kubernetes/bin/configure-helper.sh: line 661: base_metrics_memory: command not found
I will fix it immediately
yujuhong commentedon May 24, 2016
@andyzheng0831 Thanks! Assigning the issue to you for ease of tracking.
andyzheng0831 commentedon May 24, 2016
Root cause: The PR #26159 used metrics_memory="$(base_metrics_memory)", in which "(" and ")" should be "{" and "}". The script configure-helper.sh has a protective setting existing on error. So, it fails on setting up heapster and exist. The step for setting up remaining addon pods are skipped. This is intentionally, so that the breakage can be seen easily.
I am verifying my fix and then send out for review
yujuhong commentedon May 24, 2016
@andyzheng0831 where was the error printed? I also suspected that the script errored out, but wasn't sure where to look (and I didn't spend more time digging). In the case where the setup fails, it'd be great if we can surface that, instead of continue running all the tests.
9 remaining items