GCI: add support for network plugin #27027

andyzheng0831 · 2016-06-08T06:03:00Z

I had run e2e against a cluster with both master and nodes on GCI a couple of times. The PR auto tests will cover the hybrid cluster with just master on GCI.

cc/ @roberthbailey @fabioy @kubernetes/goog-image

andyzheng0831 · 2016-06-08T06:07:27Z

cluster/gce/gci/configure.sh

@@ -146,6 +146,17 @@ function install-kube-binary-config {
  else
    rm -f "${kube_bin}/kubelet"
  fi
+  if [[ "${NETWORK_PROVIDER:-}" == "kubenet" ]] || \
+     [[ "${NETWORK_PROVIDER:-}" == "cni" ]]; then
+    #TODO(andyzheng0831): We should make the cni version number as a k8s env variable.


@freehan I think we should find a way to simplify the code maintenance when bumping the cni version. One option is put the version number in a kube env variable, which will be passed to the network plugin manifest and here. So you will need to just update one place whenever you need to bump the version. What do you think?

If cni is specified then the cni version should be another kube-env field.

I am not familiar with the cni manifest change, so I leave a TODO here, and will coordinate with the cni owner freehan@ to make the change, which will be in a separate PR.

Ideally, we always want to use the latest stable cni version in kubenet. If customer want to specify their own, they can use cni network plugin and bring their own config/binary.

roberthbailey · 2016-06-08T14:48:29Z

cluster/gce/gci/configure-helper.sh

@@ -261,6 +268,12 @@ function assemble-docker-flags {
  fi

  echo "DOCKER_OPTS=\"${docker_opts} ${EXTRA_DOCKER_OPTS:-}\"" > /etc/default/docker
+  # If using a network plugin, we need to explicitly restart docker daemon, because
+  # kubelet will not do it. 
+  if [[ "${use_net_plugin=}" == "true" ]]; then


should this be "${use_net_plugin:-}"?

Nice catch, thanks! Will correct it.

andyzheng0831 · 2016-06-08T17:00:52Z

@roberthbailey @fabioy the network plugin was added since 1.2 branch but I notice currently GKE does not use the network plugin. I am not sure what is GKE's plan about this network plugin feature. If not necessary, we may avoid making a trusty change for it and cherrypick in 1.2-release branch.

freehan · 2016-06-08T17:29:35Z

cluster/gce/gci/configure-helper.sh

@@ -329,6 +343,7 @@ function start-kubelet {
      flags+=" --register-schedulable=false"
      flags+=" --reconcile-cidr=false"
      flags+=" --pod-cidr=10.123.45.0/30"
+      reconcile_cidr=false


flags += " --reconcile-cidr=false"

Ah I made a mistake that I thought the default value is false. Just check it, this flag is true by default. I will update the code

andyzheng0831 · 2016-06-08T19:38:17Z

@freehan the PR is updated, please review it. Thanks!

freehan · 2016-06-08T20:16:49Z

cluster/gce/gci/configure-helper.sh

@@ -329,6 +343,7 @@ function start-kubelet {
      flags+=" --register-schedulable=false"
      flags+=" --reconcile-cidr=false"


you may end up with duplicate flag here

This should be deleted. Will update it shortly

andyzheng0831 · 2016-06-08T21:10:41Z

@k8s-bot unit test this issue #26815

freehan · 2016-06-08T21:12:38Z

LGTM pending on tests

andyzheng0831 · 2016-06-08T21:20:00Z

@k8s-bot test this issue #IGNORE

k8s-bot · 2016-06-08T22:04:43Z

GCE e2e build/test passed for commit 66d6b43.

freehan · 2016-06-09T00:03:14Z

bump priority. This PR is required to integrate GCI with kubenet

freehan · 2016-06-09T00:25:21Z

just to be safe. Can we NOT run kubenet on master? Since we are very close to release cut.

freehan · 2016-06-09T01:00:08Z

Nevermind. just check the code once again. It should be ok.

andyzheng0831 · 2016-06-09T02:03:31Z

@freehan FYI, if you think network plugin is not ready to deploy, please set the NETWORK_PROVIDER to false. For GCI, our target is to provide feature parity as ContainerVM, and this is the reason why I made this PR to add the network plugin in GCI config. As for GKE rollout, this pr should be OK because currently GKE does not use kubenet.

k8s-github-robot · 2016-06-09T05:11:37Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2016-06-09T06:00:52Z

GCE e2e build/test passed for commit 66d6b43.

k8s-github-robot · 2016-06-09T06:01:19Z

Automatic merge from submit-queue

roberthbailey · 2016-06-09T19:19:26Z

@matchstick -- what is the plan with network plugins for 1.3?

freehan · 2016-06-09T20:53:24Z

@roberthbailey kubenet is on by default now.

Automatic merge from submit-queue Revert "Revert "GCI: add support for network plugin"" PR #27027 added the network plugin support in GCI config, but later a bug in the network plugin broke e2e tests (see issue #27118). The bug was fixed by #27141 and we have been repeatedly run the serial e2e tests more than 10 times to verify the fix. Now it should be safe to put the GCI network plugin support back. We will first merge in the master branch and monitor the Jenkins serial tests for a while and then cherry-pick it into release-1.3 branch.

bboreham · 2016-09-06T16:20:12Z

cluster/gce/gci/configure-helper.sh

@@ -341,6 +355,15 @@ function start-kubelet {
      flags+=" --hairpin-mode=${HAIRPIN_MODE}"
    fi
  fi
+  # Network plugin
+  if [[ -n "${NETWORK_PROVIDER:-}" ]]; then
+    flags+=" --network-plugin-dir=/home/kubernetes/bin"


Note that for CNI this parameter sets the directory where config is searched, not binaries. As far as I can tell CNI just doesn't work on GCI now.

True. Will fix it along with: #28563

Oh; I actually started on a PR as this is a total blocker for me.

Go ahead then. I have not started. Please also refer to the discussions in #28563.

andyzheng0831 added release-note Denotes a PR that will be considered when it comes time to generate release notes. area/os/gci labels Jun 8, 2016

andyzheng0831 added this to the v1.3 milestone Jun 8, 2016

andyzheng0831 assigned freehan Jun 8, 2016

googlebot added the cla: yes label Jun 8, 2016

andyzheng0831 reviewed Jun 8, 2016
View reviewed changes

k8s-github-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 8, 2016

roberthbailey reviewed Jun 8, 2016
View reviewed changes

freehan reviewed Jun 8, 2016
View reviewed changes

GCI: add support for kubenet

66d6b43

andyzheng0831 closed this Jun 8, 2016

andyzheng0831 reopened this Jun 8, 2016

freehan added lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jun 8, 2016

freehan removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2016

freehan added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2016

k8s-github-robot merged commit 72beb65 into kubernetes:master Jun 9, 2016

andyzheng0831 deleted the gci-network branch June 9, 2016 16:42

yujuhong mentioned this pull request Jun 9, 2016

kubernetes-e2e-gce-serial: broken test run #27118

Closed

dchen1107 mentioned this pull request Jun 9, 2016

Revert "GCI: add support for network plugin" #27147

Merged

andyzheng0831 mentioned this pull request Jun 14, 2016

Revert "Revert "GCI: add support for network plugin"" #27371

Merged

bboreham reviewed Sep 6, 2016
View reviewed changes

bboreham mentioned this pull request Sep 7, 2016

Add flag to set CNI bin dir, and use it on gci nodes #32151

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCI: add support for network plugin #27027

GCI: add support for network plugin #27027

andyzheng0831 commented Jun 8, 2016

andyzheng0831 Jun 8, 2016

roberthbailey Jun 8, 2016

andyzheng0831 Jun 8, 2016

freehan Jun 8, 2016

roberthbailey Jun 8, 2016

andyzheng0831 Jun 8, 2016

andyzheng0831 commented Jun 8, 2016

freehan Jun 8, 2016

andyzheng0831 Jun 8, 2016

andyzheng0831 commented Jun 8, 2016

freehan Jun 8, 2016

andyzheng0831 Jun 8, 2016

andyzheng0831 commented Jun 8, 2016

freehan commented Jun 8, 2016

andyzheng0831 commented Jun 8, 2016

k8s-bot commented Jun 8, 2016

freehan commented Jun 9, 2016 •

edited

Loading

freehan commented Jun 9, 2016 •

edited

Loading

freehan commented Jun 9, 2016

andyzheng0831 commented Jun 9, 2016

k8s-github-robot commented Jun 9, 2016

k8s-bot commented Jun 9, 2016

k8s-github-robot commented Jun 9, 2016

roberthbailey commented Jun 9, 2016

freehan commented Jun 9, 2016

bboreham Sep 6, 2016

freehan Sep 6, 2016

bboreham Sep 6, 2016

freehan Sep 6, 2016

		@@ -329,6 +343,7 @@ function start-kubelet {
		flags+=" --register-schedulable=false"
		flags+=" --reconcile-cidr=false"

GCI: add support for network plugin #27027

GCI: add support for network plugin #27027

Conversation

andyzheng0831 commented Jun 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyzheng0831 commented Jun 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyzheng0831 commented Jun 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyzheng0831 commented Jun 8, 2016

freehan commented Jun 8, 2016

andyzheng0831 commented Jun 8, 2016

k8s-bot commented Jun 8, 2016

freehan commented Jun 9, 2016 • edited Loading

freehan commented Jun 9, 2016 • edited Loading

freehan commented Jun 9, 2016

andyzheng0831 commented Jun 9, 2016

k8s-github-robot commented Jun 9, 2016

k8s-bot commented Jun 9, 2016

k8s-github-robot commented Jun 9, 2016

roberthbailey commented Jun 9, 2016

freehan commented Jun 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freehan commented Jun 9, 2016 •

edited

Loading

freehan commented Jun 9, 2016 •

edited

Loading