cc @wojtek-t @gmarek @andyzheng0831 @roberthbailey @mqliang

You guys have identified the root cause of the breakage? I will also revert the support from my PR #25927

We are trying to root cause it now. We have a repro case.

On Fri, May 20, 2016 at 3:25 PM, Andy Zheng notifications@github.com
wrote:

You guys have identified the root cause of the breakage? I will also
revert the support from my PR #25927
#25927

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#25997 (comment)

We haven't decided if we should merge this or try for a fix yet. @mhrgoog and @mikedanese have a repro and are looking at it right now. Other things are blocking the queue at the moment so we don't have to decide right away.

Labelling this PR as size/XL

GCE e2e build/test passed for commit 107d68d.

• Build Log
• Test Artifacts
• Internal Jenkins Results

So mike and I poked at this and I think we found the issue. After sshing to
the master node on the kubemark master we saw that the controller was not
running.

We looked at the following files.

$ cat /var/log/controller.log
I0520 22:21:43.539213 2933 plugins.go:71] No cloud provider specified.
I0520 22:21:43.539397 2933 nodecontroller.go:157] Sending events to api
server.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x7f28ab]
goroutine 1 [running]:
panic(0x2303e40, 0xc82000e100)
/usr/local/go/src/runtime/panic.go:481 +0x3e6

k8s.io/kubernetes/pkg/controller/node.(_NodeController).filterOutServiceRange(0xc820144000)
/go/src/
k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/node/nodecontroller.go:342
+0x2b
k8s.io/kubernetes/pkg/controller/node.(_NodeController).Run(0xc820144000,
0x2540be400)

...
$

The line in question is:

nodecontroller.go:342

if !nc.clusterCIDR.Contains(nc.serviceCIDR.IP.Mask(nc.clusterCIDR.Mask)) &&
!nc.serviceCIDR.Contains(nc.clusterCIDR.IP.Mask(nc.serviceCIDR.Mask)) {

Mike guessed that nc.serviceCIDR or nc.clusterCIDRwas a nil pointer.

Then we read controllers_flags

$ cat controllers_flags
--v=2__
--allocate-node-cidrs=true
--cluster-cidr=10.240.0.0/13
--terminated-pod-gc-threshold=100
$

Here he saw that SERVICE_CLUSTER_IP_RANGE is not set.

Mike figured out that this was due to a flag that is now required to be set
in order for the controller to be running was not set in the
cluster/kubemark/config-default.sh file. The flag is:
SERVICE_CLUSTER_IP_RANGE.

Once we set the flag we re-ran test/kubemark/run-e2e-tests.sh and now it is
passing.

mrubin

On Fri, May 20, 2016 at 3:33 PM, Daniel Smith notifications@github.com
wrote:

We haven't decided if we should merge this or try for a fix yet. @mhrgoog
https://github.com/mhrgoog and @mikedanese
https://github.com/mikedanese have a repro and are looking at it right
now. Other things are blocking the queue at the moment so we don't have to
decide right away.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#25997 (comment)