Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for alternative container runtime in kube-up.sh #54964

Merged
merged 1 commit into from
Nov 3, 2017
Merged

Add support for alternative container runtime in kube-up.sh #54964

merged 1 commit into from
Nov 3, 2017
+31 −4

Conversation

Random-Liu
Copy link
Member

@Random-Liu Random-Liu commented Nov 2, 2017 •
edited
Loading

For kubernetes/enhancements#286.

This PR added 4 new environment variables in kube-up.sh to support alternative container runtime:

  1. KUBE_MASTER_EXTRA_METADATA and KUBE_NODE_EXTRA_METADATA. Add extra metadata on master and node instance. With this we could specify different cloud-init for a different container runtime, and also add extra metadata for the new cloud-init, e.g. master.yaml
  2. KUBE_CONTAINER_RUNTIME_ENDPOINT. Specify different sock for different container runtime. It's only used when it's not empty.
  3. KUBE_LOAD_IMAGE_COMMAND. Specify different load image command for different container runtime.

An example for cri-containerd:

export KUBE_MASTER_EXTRA_METADATA="user-data=${GOPATH}/src/github.com/kubernetes-incubator/cri-containerd/test/e2e/master.yaml,cri-containerd-configure-sh=${GOPATH}/src/github.com/kubernetes-incubator/cri-containerd/test/configure.sh"
export KUBE_NODE_EXTRA_METADATA="user-data=${GOPATH}/src/github.com/kubernetes-incubator/cri-containerd/test/e2e/node.yaml,cri-containerd-configure-sh=${GOPATH}/src/github.com/kubernetes-incubator/cri-containerd/test/configure.sh"
export KUBE_CONTAINER_RUNTIME="remote"
export KUBE_CONTAINER_RUNTIME_ENDPOINT="/var/run/cri-containerd.sock"
export KUBE_LOAD_IMAGE_COMMAND="/home/cri-containerd/usr/local/bin/cri-containerd load"
export NETWORK_POLICY_PROVIDER="calico"

Signed-off-by: Lantao Liu lantaol@google.com

none

/cc @yujuhong @dchen1107 @feiskyer @mikebrow @abhi @mrunalp @runcom
/cc @kubernetes/sig-node-pr-reviews

@Random-Liu Random-Liu added this to the v1.9 milestone Nov 2, 2017
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. labels Nov 2, 2017
@Random-Liu Random-Liu added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Nov 2, 2017
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 2, 2017
@Random-Liu Random-Liu mentioned this pull request Nov 2, 2017
Merged
@Random-Liu Random-Liu changed the title Add support to use kube-up.sh for alternative container runtime. Add support to for alternative container runtime in kube-up.sh Nov 2, 2017
yujuhong
yujuhong reviewed Nov 2, 2017
View reviewed changes
cluster/gce/config-default.sh Outdated
RKT_VERSION=${KUBE_RKT_VERSION:-1.23.0}
RKT_STAGE1_IMAGE=${KUBE_RKT_STAGE1_IMAGE:-coreos.com/rkt/stage1-coreos}
# MASTER_EXTRA_METADATA is the extra instance metadata on master instance separated by colon.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: s/colon/colons

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.

cluster/common.sh Outdated
@@ -908,7 +911,6 @@ EOF
[[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "container-linux" ]]; then
# Container-Linux-only env vars. TODO(yifan): Make them available on other distros.
cat >>$file <<EOF
KUBERNETES_CONTAINER_RUNTIME: $(yaml-quote ${CONTAINER_RUNTIME:-rkt})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep this for backward compatibility for at least a release?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Will keep the container-linux side unchanged.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean to accept both for now :-)

cluster/gce/config-default.sh Outdated
RKT_VERSION=${KUBE_RKT_VERSION:-1.23.0}
RKT_STAGE1_IMAGE=${KUBE_RKT_STAGE1_IMAGE:-coreos.com/rkt/stage1-coreos}
# MASTER_EXTRA_METADATA is the extra instance metadata on master instance separated by colon.
MASTER_EXTRA_METADATA=${KUBE_MASTER_EXTRA_METADATA:-${KUBE_EXTRA_METADATA:-}}
# MASTER_EXTRA_METADATA is the extra instance metadata on node instance separated by colon.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: s/colon/colons

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.
Actually should be commas. Sorry.

cluster/gce/config-test.sh
@@ -80,9 +80,15 @@ NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}
NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-cos-cloud}
NODE_SERVICE_ACCOUNT=${KUBE_GCE_NODE_SERVICE_ACCOUNT:-default}
CONTAINER_RUNTIME=${KUBE_CONTAINER_RUNTIME:-docker}
CONTAINER_RUNTIME_ENDPOINT=${KUBE_CONTAINER_RUNTIME_ENDPOINT:-}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a question: when do we set KUBE_XXXXX variables?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually before we run e2e-up.sh and kube-up.sh.

@feiskyer
Copy link
Member

feiskyer commented Nov 2, 2017

Excellent work.

One small question: is KUBE_LOAD_IMAGE_COMMAND required? Loading images is not part of CRI.

@Random-Liu
Copy link
Member Author

is KUBE_LOAD_IMAGE_COMMAND required? Loading images is not part of CRI.

Yeah, in kube-up.sh we need to load image for system containers. And different container runtimes need different commands.

@Random-Liu Random-Liu force-pushed the add-containerd-e2e branch 3 times, most recently from 1b188ca to 9523fbf Compare November 2, 2017 03:42
@Random-Liu
Add containerd e2e.
75293ac 
Signed-off-by: Lantao Liu <lantaol@google.com>
@Random-Liu
Copy link
Member Author

@yujuhong Addressed comments.

@Random-Liu
Copy link
Member Author

Random-Liu commented Nov 2, 2017 •
edited
Loading

/test pull-kubernetes-bazel-test

@runcom
Copy link
Contributor

runcom commented Nov 2, 2017

Testing this out with cri-o, nice @Random-Liu

@Random-Liu
Copy link
Member Author

Testing this out with cri-o, nice @Random-Liu

You also need #54838

@yujuhong
Copy link
Contributor

yujuhong commented Nov 2, 2017

/lgtm

@Random-Liu
Copy link
Member Author

Note that this PR only added alternative container runtime for COS and Ubuntu image.
If we want this on container-linux, we also need to change the script in container-linux. /cc @euank @yifan-gu

@dchen1107
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 3, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dchen1107, Random-Liu, yujuhong

No associated issue. Update pull-request body to add a reference to an issue, or get approval with /approve no-issue

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 3, 2017
@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 54488, 54838, 54964). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit 7a28aaf into kubernetes:master Nov 3, 2017
@Random-Liu Random-Liu changed the title Add support to for alternative container runtime in kube-up.sh Add support for alternative container runtime in kube-up.sh Nov 3, 2017
@yliaog yliaog mentioned this pull request Nov 8, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yujuhong yujuhong yujuhong left review comments

Assignees

@dchen1107 dchen1107

@yujuhong yujuhong

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. sig/node Categorizes an issue or PR as relevant to SIG Node. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v1.9
Development

Successfully merging this pull request may close these issues.
7 participants
@Random-Liu @feiskyer @runcom @yujuhong @dchen1107 @k8s-github-robot @k8s-ci-robot