Add docker-in-docker based multi-node development cluster provider #26147

sttts · 2016-05-24T08:36:24Z

Followed-up by #27459

Based on cluster/mesos/docker, with all Mesos dependencies removed.

$ KUBERNETES_PROVIDER=docker-in-docker cluster/kube-up.sh

gives you a 2-node k8s cluster, completely running within Docker containers using docker-in-docker for each kubelet. Moreover, kube-dns and the dashboard are deployed.

This was originally used heavily by the Mesos Kubernetes team. Spinning up a cluster is a matter of a minute.

It's perfectly suited for e2e test trouble shooting due to multi-node support. You can even run more nodes by setting NUM_NODES.

ingvagabund · 2016-05-24T10:44:34Z

# KUBERNETES_PROVIDER=docker-in-docker cluster/kube-up.sh
... Starting cluster using provider: docker-in-docker
... calling verify-prereqs
Verifying required commands
... calling kube-up
Creating Work Dir: /root/tmp/kubernetes-dind/cluster
Pulling Docker images
Trying to pull repository docker.io/cpuguy83/docker-grand-ambassador ... 0.9.1: Pulling from cpuguy83/docker-grand-ambassador

5b7763733869: Pull complete 
6906123dd1ad: Pull complete 
f186b4a12566: Pull complete 
979ba586e38c: Pull complete 
6af51986132c: Pull complete 
95e88cbba4f3: Pull complete 
352f0b08bddb: Pull complete 
3185ff645a21: Pull complete 
2431b5c5e40f: Pull complete 
dc537da6d053: Pull complete 
44dd7a6523ff: Pull complete 
b7dcde207b0b: Pull complete 
f62eee73fe9a: Pull complete 
42a1017f8b1e: Pull complete 
f79667b5d5c1: Pull complete 
7b14707783c8: Pull complete 
Digest: sha256:b283dfcd767179af5f2f533e89f6b6ccbdd7d8df9507a663c33d10787477a548
Status: Downloaded newer image for docker.io/cpuguy83/docker-grand-ambassador:0.9.1

Trying to pull repository quay.io/coreos/etcd ... v2.2.5: Pulling from coreos/etcd

1a4aa4423d33: Pull complete 
e568c3f8d593: Pull complete 
695fa95dade5: Pull complete 
937a25e0abf8: Pull complete 
Digest: sha256:f42612de899348ba8f0e96120464b2062ea53fd4199343964f602b702c65a3b4
Status: Downloaded newer image for quay.io/coreos/etcd:v2.2.5

Using default tag: latest
Trying to pull repository k8s.io/kubernetes-dind ... not found
Error: image kubernetes-dind:latest not found

sttts · 2016-05-24T11:14:54Z

@ingvagabund that issue is fixed now.

s-urbaniak · 2016-05-24T13:00:17Z

cluster/docker-in-docker/bin/await-file

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+
+# Copyright 2015 The Kubernetes Authors All rights reserved.


This file is actually copyrighted 2015 ;-)

ingvagabund · 2016-05-24T15:59:11Z

Thanks for help @sttts, here are my steps to make it running on Fedora:

./hack/build-go.sh
dnf install docker-composer -y
modprobe overlay

Then run KUBERNETES_PROVIDER=docker-in-docker cluster/kube-up.sh.

Once you are finished with the cluster, run KUBERNETES_PROVIDER=docker-in-docker cluster/kube-down.sh

If something goes wrong on the way make sure you call kube-down.sh before you take another round. Otherwise, you can end up with outdated/invalid certificate.

luxas · 2016-05-24T19:23:11Z

cc myself, as I've been working on docker-multinode for a while

luxas · 2016-05-24T19:27:30Z

cluster/docker-in-docker/deploy-addons.sh

+bin="$(cd "$(dirname "${BASH_SOURCE}")" && pwd -P)"
+
+# create the kube-system and static-pods namespaces
+"${kubectl}" apply -f "${KUBE_ROOT}/cluster/docker-in-docker/kube-system-ns.yaml"


kube-system is created automatically with v1.3.0-alpha.4 and over: #25196

luxas · 2016-05-25T12:51:23Z

@sttts Please see #25631, how I've integrated kube-addon-manager with kube-addon-manager.
We should see what differs between the hyperkube container and maybe things between the two solutions. Also see: kubernetes-retired/kube-deploy#12

sttts · 2016-05-25T13:06:49Z

I browsed through https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/kube-addons.sh.
My first impression is that it is mainly a life-cylce tool for the addons.
Does it do templating, e.g. for the kube-dns addresses?

With cluster/docker-in-docker I would like to keep life-cycle questions out
of scope as much as possible, to avoid complexity. It's meant to be used to
launch development clusters with multiple nodes as fast as possible (=
within 1-2 minutes), e.g. for iterative development or e2e test trouble
shooting. It's not meant for long-running clusters.

With this reasoning I think the addon-manager is not really the right tool
IMO. If I misunderstand something, feel free to correct me :)

On Wed, May 25, 2016 at 2:52 PM, Lucas Käldström notifications@github.com
wrote:

@sttts https://github.com/sttts Please see #25631
#25631, how I've
integrated kube-addon-manager with kube-addon-manager.
We should see what differs between the hyperkube container and maybe
things between the two solutions. Also see: kubernetes-retired/kube-deploy#12
kubernetes-retired/kube-deploy#12

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#26147 (comment)

sttts · 2016-05-25T14:02:24Z

@k8s-bot test this issue: #22409

sttts · 2016-05-27T10:19:03Z

@ingvagabund I have added your instructions to docs/devel/running-locally.md. Also you don't have to call kube-down.sh anymore. Last but not least, it will tell you about missing overlay/aufs support.

ingvagabund · 2016-05-27T11:55:56Z

@sttts thanks, now I can call that user friendly :)

mikedanese · 2016-05-27T16:39:54Z

https://github.com/kubernetes/kube-deploy is a better repo for this work. We are trying to remove cluster/ from the main repo.

sttts · 2016-05-27T16:51:37Z

@mikedanese Does this mean that the cluster/ scripts (e.g. for e2e tests) will migrate as well? The kube-deploy repo still looks pretty "empty" in that regard. This docker-in-docker cluster is not meant as a "real" deployment for v1.x release versions, but as an extension of hack/local-up-cluster.sh purely for development, especially of e2e tests. It closely follows master and should work with master all the time, or with any other version if you want to test something in v1.x.y. IMO the other repo doesn't sound like the right place for that.

sttts · 2016-05-27T16:53:27Z

Having said that, I am happy to split the docker-in-docker code from the cluster directory and turn it into a self-contained script if that helps. It doesn't use much infrastructure from there anyway.

mikedanese · 2016-05-27T17:05:11Z

Our goal is to split out cluster/ into a separate repo, kube-deploy and people are actively working towards this, e.g. #26031. It's going to be a process that takes time, but this has very little dependency on anything in cluster so I don't see a very good reason to add it there temporarily when we can put it in a proper place on the get go. I'm going to push back on all new automation that are proposed in cluster/ and declare maintenance mode.

We are discussing this in #23174

sttts · 2016-05-27T17:15:34Z

@mikedanese Sounds all reasonable for clusters which are used with named release versions. My fear is that you always have to search the right revision in the kube-deploy repo which happens to use the right command line flags to work with the kubernetes/kubernetes repo at hand. I wonder how you solve the same problem with e2e tests on GCE or in the vagrant cluster. Will read through #23174, maybe my concerns are solved already.

sttts · 2016-05-30T06:46:15Z

@mikedanese having read through #23174 I still feel that this PR is misplaced in the discussion of cluster v2 in #23174. It mostly fits into what @bgrant0607 calls (1) in #23174 (comment): purely for development and without any machine to ssh/ansible into, i.e.

(0) hack/local-up-cluster < (0+) docker-in-docker < (1) minikube < (2) good enough cluster ....

Having said that, I think there is no proper home for this at the moment if cluster/ is deprecated. I can move this PR into its own repo or into an independent directory of kube-deploy. Either way makes it harder to use and to find for new developers. It would probably end up like

$ cd kubernetes
$ git clone git@github.com:some/repo/docker-in-docker docker-in-docker
$ docker-in-docker/up.sh

That workflow is fine for my use-case.

bgrant0607 · 2016-05-31T02:56:21Z

cc @vishh @dlorenc

cheld · 2016-05-31T12:38:37Z

CC @batikanu, @zreigz, @f-higashi, @taimir

... into a pure Kubernetes cluster: KUBERNETES_PROVIDER=docker-in-docker cluster/kube-up.sh will launch a 2-node cluster running completely in Docker.

k8s-bot · 2016-06-02T12:33:34Z

GCE e2e build/test passed for commit 952e01e.

piosz · 2016-06-15T18:35:52Z

@sttts you pushed this change to the main repo rather than to you private fork. Could you please close it, remove the branch and then work in you private repo?

sttts · 2016-06-15T18:39:29Z

@piosz will do

sttts · 2016-06-15T18:44:55Z

Followed-up by #27459

piosz · 2016-06-15T18:49:42Z

@sttts thanks. BTW you can invoke something like this git remote set-url --push upstream no_push to avoid further pushes to the main repo.

sttts added the team/cluster label May 24, 2016

googlebot added the cla: yes label May 24, 2016

k8s-github-robot assigned roberthbailey May 24, 2016

k8s-github-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. release-note-label-needed labels May 24, 2016

sttts added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed labels May 24, 2016

s-urbaniak reviewed May 24, 2016
View reviewed changes

luxas reviewed May 24, 2016
View reviewed changes

roberthbailey assigned mikedanese and unassigned roberthbailey May 24, 2016

sttts force-pushed the sttts-docker-compose branch from 99be434 to eadd8ad Compare May 25, 2016 08:57

sttts mentioned this pull request May 30, 2016

Return container command exit codes in kubectl run/exec #26541

Merged

3 tasks

sttts force-pushed the sttts-docker-compose branch from 948fbc7 to 0420a2c Compare May 31, 2016 11:51

sttts and others added 3 commits June 1, 2016 13:20

Initial copy cluster/{mesos -> .}/docker

4654692

Port cluster/docker-in-docker away from Mesos

53ebf13

... into a pure Kubernetes cluster: KUBERNETES_PROVIDER=docker-in-docker cluster/kube-up.sh will launch a 2-node cluster running completely in Docker.

Fix docker-in-docker cluster with new kube-dns

2aac3f6

sttts force-pushed the sttts-docker-compose branch from 0420a2c to 2aac3f6 Compare June 1, 2016 11:20

Remove debug output

952e01e

sttts mentioned this pull request Jun 15, 2016

Add docker-in-docker based multi-node hack/dind-up-cluster.sh #27459

Closed

sttts closed this Jun 15, 2016

sttts deleted the sttts-docker-compose branch June 15, 2016 18:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add docker-in-docker based multi-node development cluster provider #26147

Add docker-in-docker based multi-node development cluster provider #26147

sttts commented May 24, 2016 •

edited

Loading

ingvagabund commented May 24, 2016

sttts commented May 24, 2016

s-urbaniak May 24, 2016

sttts May 24, 2016

ingvagabund commented May 24, 2016 •

edited

Loading

luxas commented May 24, 2016

luxas May 24, 2016

sttts May 25, 2016

luxas commented May 25, 2016

sttts commented May 25, 2016

sttts commented May 25, 2016

sttts commented May 27, 2016 •

edited

Loading

ingvagabund commented May 27, 2016

mikedanese commented May 27, 2016 •

edited

Loading

sttts commented May 27, 2016

sttts commented May 27, 2016

mikedanese commented May 27, 2016

sttts commented May 27, 2016

sttts commented May 30, 2016

bgrant0607 commented May 31, 2016

cheld commented May 31, 2016

k8s-bot commented Jun 2, 2016

piosz commented Jun 15, 2016

sttts commented Jun 15, 2016

sttts commented Jun 15, 2016

piosz commented Jun 15, 2016 •

edited

Loading

		@@ -0,0 +1,42 @@
		#!/usr/bin/env bash

		# Copyright 2015 The Kubernetes Authors All rights reserved.

Add docker-in-docker based multi-node development cluster provider #26147

Add docker-in-docker based multi-node development cluster provider #26147

Conversation

sttts commented May 24, 2016 • edited Loading

Followed-up by #27459

ingvagabund commented May 24, 2016

sttts commented May 24, 2016

s-urbaniak May 24, 2016

Choose a reason for hiding this comment

sttts May 24, 2016

Choose a reason for hiding this comment

ingvagabund commented May 24, 2016 • edited Loading

luxas commented May 24, 2016

luxas May 24, 2016

Choose a reason for hiding this comment

sttts May 25, 2016

Choose a reason for hiding this comment

luxas commented May 25, 2016

sttts commented May 25, 2016

sttts commented May 25, 2016

sttts commented May 27, 2016 • edited Loading

ingvagabund commented May 27, 2016

mikedanese commented May 27, 2016 • edited Loading

sttts commented May 27, 2016

sttts commented May 27, 2016

mikedanese commented May 27, 2016

sttts commented May 27, 2016

sttts commented May 30, 2016

bgrant0607 commented May 31, 2016

cheld commented May 31, 2016

k8s-bot commented Jun 2, 2016

piosz commented Jun 15, 2016

sttts commented Jun 15, 2016

sttts commented Jun 15, 2016

piosz commented Jun 15, 2016 • edited Loading

sttts commented May 24, 2016 •

edited

Loading

ingvagabund commented May 24, 2016 •

edited

Loading

sttts commented May 27, 2016 •

edited

Loading

mikedanese commented May 27, 2016 •

edited

Loading

piosz commented Jun 15, 2016 •

edited

Loading