Make secrets at cluster startup. #5470

erictune · 2015-03-13T22:35:34Z

zmerlynn · 2015-03-14T00:28:28Z

Whatever you're trying to do with that script, it needs to go in kube-addons instead, because I'm going to bounce out anything that's at the tail of kube-up.sh. (I guess I need to get around to publishing the GKE manifesto, but #3579 is part of it.) Also, if you're generating variables like this, it'll require GKE side changes, too, so we need to talk.

erictune · 2015-03-16T14:14:09Z

Is it a kube-addon if it is an absolutely essential component of a k8s cluster?

zmerlynn · 2015-03-16T14:43:53Z

Good question on naming, but the original intent of having it created on the master still holds. I was originally thinking of calling these things "system objects" and not "addon objects", so maybe this would precipitate that change.

erictune · 2015-03-16T15:03:29Z

@ArtfulCoder

The pod json/yaml for the scheduler and the controller-manager should go into, say:
cluster/addons/scheduler and cluster/addons/controller-manager.
Then near the top of cluster/saltbase/salt/kube-addons/init.sls, add a file.recurse to copy those files to the master via salt. Make it unconditional.

I don't believe that any change is needed in cluster/saltbase/salt/kube-addons/kube-addons.sh.

Right now, there is no ordering between the different addon objects. This should still "just work". That is, create all the object in any order. Without a scheduler, none of the objects will start except the scheduler, which has a Host field set. Without a controller-manager, none of the replication controllers will run, until it does. Then they should just work. Etc.

@zmerlynn tell me if above is correct.

erictune · 2015-03-16T15:13:52Z

The question is how to get the variables we need (tokens) from kube-up to kube-addons.

Option 1: In the kube-addons.sh script, read the tokens from the known_tokens.csv file, and then create secret objects for each one. Pros: less plumbing. Cons: requires full filesystem access for the `kube-addons.sh script. I think it has this

Option 2: Actually generates the tokens in kube-addons.sh script, and append them to known_tokens.csv at that point. Pros: less shipping-around of secrets. Cons: Have to restart apiserver to pick up the new tokens.

zmerlynn · 2015-03-16T17:20:07Z

The part where you're adding the env variable and generation of the token to kube-up.sh is fine. I'd actually prefer keeping that out of kube-addons.sh.

Why do you need two more tokens, btw?

erictune · 2015-03-16T18:11:41Z

I want each system component to have its own identity so each component can have only the privileges it needs.

erictune · 2015-03-27T23:10:49Z

@zmerlynn I've moved most of my changes into kube-addons.sh

@ArtfulCoder you can use these secrets to authorize scheduler and replication controller.

@vishh you can use these secrets to authorize monitoring pods that need to talk to the apiserver.

erictune · 2015-03-27T23:12:47Z

cluster/gce/configure-vm.sh

+  for a in scheduler controller_manager logging monitoring dns
+  do
+    token=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)
+    echo "${token},kubernetes_builtin:$a,kubernetes_builtin:$a" >> "${known_tokens_file}"


@smarterclayton ISTR you suggesting a format for namespacing usernames but I can't find the issue now. Is there already a standard used in openshift? Here I used kubernetes_builtin:$USER, but I am open to whatever.

@liggitt

We ended up with normal usernames being unnamedspaced (so they can inherit the preferred username from the bedrock identity provider), and namespaced system/service account usernames with a "prefix:" (like "system:" or "project:"). That means ":" is disallowed in normal usernames for us.

We had very specific users names and we are using the system: prefix to scope deeper roles.

https://github.com/openshift/origin/blob/master/pkg/cmd/server/bootstrappolicy/constants.go
https://github.com/openshift/origin/blob/master/pkg/cmd/server/admin/default_certs.go

Would be good to settle on a pattern for prefixing as we go (don't have to change here).

On Mar 27, 2015, at 7:18 PM, Jordan Liggitt notifications@github.com wrote:

In cluster/gce/configure-vm.sh:

@@ -260,6 +260,13 @@ function create-salt-auth() {
kubelet_auth_file="/srv/salt-overlay/salt/kubelet/kubernetes_auth"
(umask 077;
echo "{"BearerToken": "${KUBELET_TOKEN}", "Insecure": true }" > "${kubelet_auth_file}")
+

Generate tokens for other "service accounts". Append to known_tokens.

for a in scheduler controller_manager logging monitoring dns

do

token=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)

echo "${token},kubernetes_builtin:$a,kubernetes_builtin:$a" >> "${known_tokens_file}"
We ended up with normal usernames being unnamedspaced (so they can inherit the preferred username from the bedrock identity provider), and namespaced system/service account usernames with a "prefix:" (like "system:" or "project:"). That means ":" is disallowed in normal usernames for us.

—
Reply to this email directly or view it on GitHub.

erictune · 2015-03-27T23:15:20Z

A step to fixing #5921

zmerlynn · 2015-03-28T00:38:26Z

This looks like it should work and actually require no GKE side changes.

vishh · 2015-03-28T03:28:21Z

@erictune: Ack. I will update heapster as soon as this PR is merged.

erictune · 2015-04-02T18:07:50Z

@zmerlynn would you mind reviewing?

zmerlynn · 2015-04-02T18:31:06Z

Happy to.

zmerlynn · 2015-04-02T18:35:09Z

cluster/saltbase/salt/kube-addons/kube-addons.sh

+  obj="$1"
+
+  for tries in {1..5}; do
+    if ( echo "${obj}" | ${KUBECTL} --server="127.0.0.1:8080" create -f - ); then


Any reason not to pass --validate here as well?

ok. there was a bug in kubectl with validate and some objects, but it is since fixed.

pmorie · 2015-04-02T18:42:06Z

@liggitt This seems relevant to your interests

zmerlynn · 2015-04-02T18:43:31Z

cluster/saltbase/salt/kube-addons/kube-addons.sh

@@ -19,6 +19,39 @@
 # managed result is of that. Start everything below that directory.
 KUBECTL=/usr/local/bin/kubectl

+function create-kubernetesauth-secret() {


Blah, I didn't enforce it well in create-object, but you can fix it while you're in here. :)

These should all be local. Probably local -r token=$1, etc, since they look readonly.

zmerlynn · 2015-04-02T18:47:00Z

cluster/gce/configure-vm.sh

+  for a in "system:scheduler" "system:controller_manager" "system:logging" "system:monitoring" "system:dns"
+  do
+    token=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)
+    echo "${token},$a,$a" >> "${known_tokens_file}"


Nit: I try to follow ${foo} for all variables. (so ${a}. Or ${account}). And below.

zmerlynn · 2015-04-02T18:47:58Z

I did my best to channel @filbranden. The actual code LGTM, it's just style nits.

erictune · 2015-04-02T21:39:01Z

Needs rebase onto #6384

These secrets will be used in subsequent PRs by: scheduler, controller-manager, monitoring services, logging services, and skydns. Each of these services will then be able to stop using kubernetes-ro or host networking.

erictune · 2015-04-02T23:10:07Z

Okay, rebased. Testing.

erictune · 2015-04-02T23:15:13Z

Okay, kube-up.sh works, and expected addons and new secrets are created.

zmerlynn · 2015-04-03T00:43:52Z

LGTM.

Make secrets at cluster startup.

erictune · 2015-04-03T01:06:12Z

Thanks.

Changes approach taken in kubernetes#5470 Instead of creating a kubernetes_auth file, which we are tring to get away from, creates a kubeconfig file, which is the new hotness. Instead of creating the kubeconfig file in the kube-addon script on the master, it creates it at the time of salt-overlay generation. More information is available at this time. In particular, the master certs and master address are handy at this point, so those are included in the kubeconfig file. The kube-addons script is simplified because the secret objects are now just plain olf yaml files, which it knows how to create, just like it creates pods and services. Kubectl is used to generate the kubeconfig file. This ensures correct format and is more self-documenting, and matches how the admin credentials are done. TODO(erictune): do this for kubelet.

googlebot added the cla: yes label Mar 13, 2015

erictune assigned ArtfulCoder Mar 13, 2015

erictune force-pushed the for-abhis branch from d762e7c to 78ac96f Compare March 27, 2015 23:08

erictune changed the title ~~WIP: Sketch of making scheduler from pod with secret~~ Make secrets at cluster startup. Mar 27, 2015

erictune reviewed Mar 27, 2015
View reviewed changes

erictune mentioned this pull request Mar 30, 2015

Recommended way to access r/w apiserver from container? #6186

Closed

erictune force-pushed the for-abhis branch 2 times, most recently from 252a66d to aa91596 Compare April 2, 2015 17:53

erictune assigned zmerlynn and unassigned ArtfulCoder Apr 2, 2015

zmerlynn reviewed Apr 2, 2015
View reviewed changes

erictune force-pushed the for-abhis branch 3 times, most recently from 930e8d6 to 59b3868 Compare April 2, 2015 22:58

Make secrets at cluster startup.

59daeab

These secrets will be used in subsequent PRs by: scheduler, controller-manager, monitoring services, logging services, and skydns. Each of these services will then be able to stop using kubernetes-ro or host networking.

erictune force-pushed the for-abhis branch from 59b3868 to 59daeab Compare April 2, 2015 23:01

zmerlynn added a commit that referenced this pull request Apr 3, 2015

Merge pull request #5470 from erictune/for-abhis

12cf768

Make secrets at cluster startup.

zmerlynn merged commit 12cf768 into kubernetes:master Apr 3, 2015

erictune mentioned this pull request Apr 17, 2015

Securing kubernetes-ro use cases #5921

Closed

erictune mentioned this pull request Apr 17, 2015

Make service-account secrets in kubectl form. #7011

Closed

WIZARD-CXY mentioned this pull request May 7, 2015

How does bare metal machine use token #7892

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make secrets at cluster startup. #5470

Make secrets at cluster startup. #5470

erictune commented Mar 13, 2015

zmerlynn commented Mar 14, 2015

erictune commented Mar 16, 2015

zmerlynn commented Mar 16, 2015

erictune commented Mar 16, 2015

erictune commented Mar 16, 2015

zmerlynn commented Mar 16, 2015

erictune commented Mar 16, 2015

erictune commented Mar 27, 2015

erictune Mar 27, 2015

liggitt Mar 27, 2015

smarterclayton Mar 28, 2015

Generate tokens for other "service accounts". Append to known_tokens.

erictune commented Mar 27, 2015

zmerlynn commented Mar 28, 2015

vishh commented Mar 28, 2015

erictune commented Apr 2, 2015

zmerlynn commented Apr 2, 2015

zmerlynn Apr 2, 2015

erictune Apr 2, 2015

pmorie commented Apr 2, 2015

zmerlynn Apr 2, 2015

erictune Apr 2, 2015

zmerlynn Apr 2, 2015

erictune Apr 2, 2015

zmerlynn commented Apr 2, 2015

erictune commented Apr 2, 2015

erictune commented Apr 2, 2015

erictune commented Apr 2, 2015

zmerlynn commented Apr 3, 2015

erictune commented Apr 3, 2015

Make secrets at cluster startup. #5470

Make secrets at cluster startup. #5470

Conversation

erictune commented Mar 13, 2015

zmerlynn commented Mar 14, 2015

erictune commented Mar 16, 2015

zmerlynn commented Mar 16, 2015

erictune commented Mar 16, 2015

erictune commented Mar 16, 2015

zmerlynn commented Mar 16, 2015

erictune commented Mar 16, 2015

erictune commented Mar 27, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Generate tokens for other "service accounts". Append to known_tokens.

erictune commented Mar 27, 2015

zmerlynn commented Mar 28, 2015

vishh commented Mar 28, 2015

erictune commented Apr 2, 2015

zmerlynn commented Apr 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmorie commented Apr 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zmerlynn commented Apr 2, 2015

erictune commented Apr 2, 2015

erictune commented Apr 2, 2015

erictune commented Apr 2, 2015

zmerlynn commented Apr 3, 2015

erictune commented Apr 3, 2015