Secrets proposal #4126

pmorie · 2015-02-04T21:00:12Z

A proposal for the distribution of secrets (passwords, keys, etc) to containers inside Kubernetes using a custom volume type.

roberthbailey · 2015-02-04T21:18:59Z

/cc @erictune

smarterclayton · 2015-02-04T21:30:02Z

This is subordinate to #4029 - high level discussion there (argument about the linkage between container security context, service account, and secrets). Also see the original service accounts proposal (linked over there).

markturansky · 2015-02-04T22:33:06Z

How do the secrets get onto the host by the volume plugin? It seems implied that the secrets would be read from a store by the plugin and written to local disk? No actual watching pods or attaching required. The plugin is a pass through to the secret data?

pmorie · 2015-02-05T00:48:44Z

@markturansky Your reading of the implication is correct (at least in this iteration of our design); the volume plugin is responsible for fetching the needed secrets and writing them to the local disk.

erictune · 2015-02-05T01:43:47Z

Looking forward to giving this a close read on monday.
On Feb 4, 2015 4:49 PM, "Paul Morie" notifications@github.com wrote:

@markturansky https://github.com/markturansky Your reading of the
implication is correct (at least in this iteration of our design); the
volume plugin is responsible for fetching the needed secrets and writing
them to the local disk.

—
Reply to this email directly or view it on GitHub
#4126 (comment)
.

markturansky · 2015-02-05T14:59:37Z

@pmorie so, do you really need a file or do you actually want a stream? The volume plugin would need to read the secrets from somewhere, too. Why write to disk at all?

markturansky · 2015-02-05T15:01:51Z

@pmorie nevermind, i see you want the various tools to read from those config files. And the volume plugin would clean up after itself, like EmptyDir.

pmorie · 2015-02-05T19:14:00Z

@markturansky Do you think that should be made more clear in the proposal?

markturansky · 2015-02-05T20:06:14Z

It's interesting that you want to make it a volume, because i'm looking into that same stuff right now. There seem to be strong parallels with what I'm doing.

We both want to:

Allow a user to "Define something that persists that I can mount"
Allow me to put my secret data in there ('secrets' and my MySQL database are both secret to our namespaces)
Allow that data to live longer than my pod and follow the pod around the cluster.
Your data is read-only many

markturansky · 2015-02-06T15:23:58Z

docs/design/secrets.md

+2.  As a user, I want to allow containers to consume supplemental information about services such
+    as username and password which should be kept secret, so that I can share secrets about a
+    service amongst the containers in my application securely
+


Are operator secrets global? Meaning, is that one .dockercfg for all pods or do you intend to allow each pod to have their own .dockercfg?

If global, would would you schedule that resource onto all hosts exactly once? If on a pod basis, how would you inject that operator secret volume into the pods?

On Feb 6, 2015, at 10:24 AM, Mark Turansky notifications@github.com wrote:

In docs/design/secrets.md:

+## Use Cases
+
+1. As a user, I want to store secret artifacts for my applications and consume them securely in

containers, so that I can keep the configuration for my applications separate from the images

that use them:

As a cluster operator, I want to allow a pod to access the Kubernetes master using a custom

`.kubeconfig` file, so that I can securely reach the master

As a cluster operator, I want to allow a pod to access a Docker registry using credentials

from a `.dockercfg` file, so that containers can push images

As a cluster operator, I want to allow a pod to access a git repository using SSH keys,

so that I can push and fetch to and from the repository
+2. As a user, I want to allow containers to consume supplemental information about services such

as username and password which should be kept secret, so that I can share secrets about a

service amongst the containers in my application securely

Are operator secrets global? Meaning, is that one .dockercfg for all pods or do you intend to allow each pod to have their own .dockercfg?

Both. Pods carry with them authorization info about the ability to pull images.
If global, would would you schedule that resource onto all hosts exactly once? If on a pod basis, how would you inject that operator secret volume into the pods?

—
Reply to this email directly or view it on GitHub.

IIUC, this Proposal doesn't yet say how you would use this for a .dockercfg. Kubelet would need to:

pull the secret with the per-pod dockercfg

merge that into the actual system dockercfg

pull the pod's images

somehow make sure that other pods that should not have access to that secret do use it for their images via some sort of race condition.

somehow make sure that other pods that should not have access to that secret do use it for their images via some sort of race condition.

@erictune confirm that you meant "don't" instead of "do" here?

i meant don't

@erictune Reading over this again and discussing it with @ncdc, I wonder if there's some stream-crossing in the document. In this case I'm referring to a use-case in OpenShift of wanting to push to a docker registry, rather than authorizing the kubelet to use a .dockercfg contained in a secret to docker pull an image to run in a pod.

I did a quick look for the use case I described above (kubelet uses a secret as credential for docker pull) and couldn't find anything which I felt already captured it, though it's hinted at in:

#2209
#499

Do we need to call this out as a use-case to solve with secrets @erictune @smarterclayton ?

On Feb 12, 2015, at 8:15 PM, Paul Morie notifications@github.com wrote:

In docs/design/secrets.md:

+## Use Cases
+
+1. As a user, I want to store secret artifacts for my applications and consume them securely in

containers, so that I can keep the configuration for my applications separate from the images

that use them:

As a cluster operator, I want to allow a pod to access the Kubernetes master using a custom

`.kubeconfig` file, so that I can securely reach the master

As a cluster operator, I want to allow a pod to access a Docker registry using credentials

from a `.dockercfg` file, so that containers can push images

As a cluster operator, I want to allow a pod to access a git repository using SSH keys,

so that I can push and fetch to and from the repository
+2. As a user, I want to allow containers to consume supplemental information about services such

as username and password which should be kept secret, so that I can share secrets about a

service amongst the containers in my application securely

@erictune Reading over this again and discussing it with @ncdc, I wonder if there's some stream-crossing in the document. In this case I'm referring to a use-case in OpenShift of wanting to push to a docker registry, rather than authorizing the kubelet to use a .dockercfg contained in a secret to docker pull an image to run in a pod.

I did a quick look for the use case I described above (kubelet uses a secret as credential for docker pull) and couldn't find anything which I felt already captured it, though it's hinted at in:

#2209
#499

Do we need to call this out as a use-case to solve with secrets @erictune @smarterclayton ?

Yes, we discussed this today. It's two different use cases, pull from an integrated registry is an Openshift and CoreOS problem, and it's generally useful to the community.
—
Reply to this email directly or view it on GitHub.

markturansky · 2015-02-07T02:21:42Z

I think I know why this overlaps so much. Data are secrets or secrets are data.

The secret volume plugin would implement the actual retrieval and laying down of secrets on the Kubelet's file system. Secrets may be stored in a special secret registry, as Docker volumes, LDAP etc.

Many data stores could be supported by implementing many volume plugins, as you are doing with a secret volume plugin. All secrets are pets, so a plugin could support any means of obtaining that data.

If that is true, then the operator can describe all his secrets as a set of volumes (secret volume or otherwise) in yaml or json.

The user wanting to store a secret has to request a resource (the secret volume), put the secret in the volume, and then reference that volume in the pod.

smarterclayton · 2015-02-07T03:01:03Z

Also, any data written to a disk is done in a security context (a Linux user uid and mcs labels), and for a disk to be reused it needs to be within the same security context or a permission change needs to occur. All disk sharing must be done within a single security context (the underlying uid when user namespaces are in use).

On Feb 6, 2015, at 9:21 PM, Mark Turansky notifications@github.com wrote:

I think I know why this overlaps so much. Data are secrets or secrets are data.

The secret volume plugin would implement the actual retrieval and laying down of secrets on the Kubelet's file system. Secrets may be stored in a special secret registry, as Docker volumes, LDAP etc.

Many data stores could be supported by implementing many volume plugins, as you are doing with a secret volume plugin. All secrets are pets, so a plugin could support any means of obtaining that data.

If that is true, then the operator can describe all his secrets as a set of volumes (secret volume or otherwise) in yaml or json.

The user wanting to store a secret has to request a resource (the secret volume), put the secret in the volume, and then reference that volume in the pod.

—
Reply to this email directly or view it on GitHub.

markturansky · 2015-02-07T15:11:03Z

@smarterclayton wouldn't those security features be similarly useful in persistent volumes?

So long as secrets (from any store) are expressed as volumes, couldn't this entire feature be implemented using persistent volumes w/ the additional security you mention?

The only piece missing is the once-per-host volume, which I'm sure is easily accomplished because there is, i believe, a means of having the cluster run one type of pod per host for administration purposes.

There is so much overlap here that I wonder about the design of this one (or the other one). They seem essentially the same, though, just with different nouns, but both nouns (data and secrets) are both still secrets.

smarterclayton · 2015-02-07T23:35:16Z

On Feb 7, 2015, at 10:11 AM, Mark Turansky notifications@github.com wrote:

@smarterclayton wouldn't those security features be similarly useful in persistent volumes?

What I was describing was for persistent volumes.
So long as secrets (from any store) are expressed as volumes, couldn't this entire feature be implemented using persistent volumes w/ the additional security you mention?

The use cases for persistent volumes and secrets aren't the same. They have lots of similarities, which is good, but I don't think they are used the same way or users experience them the same way.
The only piece missing is the once-per-host volume, which I'm sure is easily accomplished because there is, i believe, a means of having the cluster run one type of pod per host for administration purposes.

I don't think once per host volumes are the key part of secrets, in fact they are an implementation detail.
There is so much overlap here that I wonder about the design of this one (or the other one). They seem essentially the same, though, just with different nouns, but both nouns (data and secrets) are both still secrets.

—
Reply to this email directly or view it on GitHub.

pmorie · 2015-02-08T00:19:46Z

I agree with @smarterclayton. There are a lot of similarities but there are
also significant differences in user experience. For example, I don't see
much of a use case to store content in a persistent volume via the REST
api, but that is definitely how users will expect to create secrets.
On Sat, Feb 7, 2015 at 6:35 PM Clayton Coleman notifications@github.com
wrote:

On Feb 7, 2015, at 10:11 AM, Mark Turansky notifications@github.com
wrote:

@smarterclayton wouldn't those security features be similarly useful in
persistent volumes?

What I was describing was for persistent volumes.
So long as secrets (from any store) are expressed as volumes, couldn't
this entire feature be implemented using persistent volumes w/ the
additional security you mention?

The use cases for persistent volumes and secrets aren't the same. They
have lots of similarities, which is good, but I don't think they are used
the same way or users experience them the same way.
The only piece missing is the once-per-host volume, which I'm sure is
easily accomplished because there is, i believe, a means of having the
cluster run one type of pod per host for administration purposes.

I don't think once per host volumes are the key part of secrets, in fact
they are an implementation detail.
There is so much overlap here that I wonder about the design of this one
(or the other one). They seem essentially the same, though, just with
different nouns, but both nouns (data and secrets) are both still secrets.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#4126 (comment)
.

smarterclayton · 2015-02-08T01:17:38Z

We should catalog similarities, but not get hung up on perfectly aligning them.

For example, we will eventually want to generate secrets on demand for containers (mint a .kubeconfig that is automatically refreshed on demand, mint private keys that can be shared to all pods in a pool), and persistent volumes may have some use cases for pre generated data, but the problem domains really don't overlap.

On Feb 7, 2015, at 7:20 PM, Paul Morie notifications@github.com wrote:

I agree with @smarterclayton. There are a lot of similarities but there are
also significant differences in user experience. For example, I don't see
much of a use case to store content in a persistent volume via the REST
api, but that is definitely how users will expect to create secrets.
On Sat, Feb 7, 2015 at 6:35 PM Clayton Coleman notifications@github.com
wrote:

On Feb 7, 2015, at 10:11 AM, Mark Turansky notifications@github.com
wrote:

@smarterclayton wouldn't those security features be similarly useful in
persistent volumes?

What I was describing was for persistent volumes.
So long as secrets (from any store) are expressed as volumes, couldn't
this entire feature be implemented using persistent volumes w/ the
additional security you mention?

The use cases for persistent volumes and secrets aren't the same. They
have lots of similarities, which is good, but I don't think they are used
the same way or users experience them the same way.
The only piece missing is the once-per-host volume, which I'm sure is
easily accomplished because there is, i believe, a means of having the
cluster run one type of pod per host for administration purposes.

I don't think once per host volumes are the key part of secrets, in fact
they are an implementation detail.
There is so much overlap here that I wonder about the design of this one
(or the other one). They seem essentially the same, though, just with
different nouns, but both nouns (data and secrets) are both still secrets.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#4126 (comment)
.

—
Reply to this email directly or view it on GitHub.

markturansky · 2015-02-08T14:58:40Z

Different users manage secrets than persistent volumes. Agreed, secrets should not be implemented as volumes.

SecretSource is the common denominator. If you want to adapt the secret to a volume, we have a common scheduling problem: getting the right volumes to the right hosts and pods.

Or perhaps put another way, the volume controller I'm making for scheduling would intrinsically be able to schedule your volumes. All you need to do is put a claim on it.

We might be able to handle this distribution of volumes to hosts (the scheduling problem) the same way.

smarterclayton · 2015-02-10T19:55:26Z

I don't see how secrets are a scheduling problem?

----- Original Message -----

Different users manage secrets than persistent volumes. Agreed, secrets
should not be implemented as volumes.

SecretSource is the common denominator. If you want to adapt the secret to a
volume, we have a common scheduling problem: getting the right volumes to
the right hosts and pods.

Or perhaps put another way, the volume controller I'm making for scheduling
would intrinsically be able to schedule your volumes. All you need to do is
put a claim on it.

We might be able to handle this distribution of volumes to hosts (the
scheduling problem) the same way.

Reply to this email directly or view it on GitHub:
#4126 (comment)

markturansky · 2015-02-10T20:15:22Z

I'm sure you're right. Secrets, by definition, are pets and so are volumes. I took matching pets to hosts/pods as the common scheduling problem/pattern.

----- Original Message -----
From: "Clayton Coleman" notifications@github.com
To: "GoogleCloudPlatform/kubernetes" kubernetes@noreply.github.com
Cc: "Mark Turansky" mturansk@redhat.com
Sent: Tuesday, February 10, 2015 2:55:55 PM
Subject: Re: [kubernetes] WIP: Secrets proposal / prototype (#4126)

I don't see how secrets are a scheduling problem?

----- Original Message -----

Different users manage secrets than persistent volumes. Agreed, secrets
should not be implemented as volumes.

SecretSource is the common denominator. If you want to adapt the secret to a
volume, we have a common scheduling problem: getting the right volumes to
the right hosts and pods.

Or perhaps put another way, the volume controller I'm making for scheduling
would intrinsically be able to schedule your volumes. All you need to do is
put a claim on it.

We might be able to handle this distribution of volumes to hosts (the
scheduling problem) the same way.

Reply to this email directly or view it on GitHub:
#4126 (comment)

Reply to this email directly or view it on GitHub:
#4126 (comment)

pmorie · 2015-02-11T04:59:50Z

Rebased into a single commit for ease of rebase.

erictune · 2015-02-11T17:42:01Z

docs/design/secrets.md

+In use cases for CRUD operations, the user experience for secrets should be no different than for
+other API resources.
+
+TODO: document possible relation with Service Accounts


and Security Contexts #3910

EricMountain-1A · 2015-02-17T10:18:37Z

docs/design/secrets.md

+#### Example: service account consumes auth token secret
+
+As an example, the service account proposal discusses service accounts consuming secrets which
+contain kubernetes auth tokens.  When a Kubelet starts a pod associates with a service account


s/starts a pod associates/starts a pod that associates/ ?

pmorie · 2015-02-17T22:52:40Z

I think it should be a validation error if you declare a VolumeMount with readonly = false on a secret volume, thoughts @EricMountain-1A @erictune @smarterclayton

erictune · 2015-02-18T00:11:52Z

docs/design/secrets.md

+
+For now, we will not implement validations around these limits.  Cluster operators will decide how
+much node storage is allocated to secrets. It will be the operator's responsibility to ensure that
+the allocated storage is sufficient for the workload scheduled onto a node.


The other option is for kubelet to add up the size of all the secrets files as it writes them to tmpfs, and then to subtract this total from the memory limit that it writes for the container. This makes most or all of the limits mentioned above go away.

However, IIRC we only have per-container memory limits and not per-pod memory limits, at present. So, either you have to decide how to apportion the shortfall across the various containers (proportional?), or else charge it to the pod as a whole, which means the scheduler can't see the charge at scheduling time.

erictune · 2015-02-18T00:50:51Z

The design doc LGTM. Happy to merge that as is.

The new volume type definition also looks good.

Are you ready to have the volume plugin code reviewed? I'm fine with reviewing what you have and then working on the /tmpfs stuff in a separate PR.

erictune · 2015-02-18T00:52:44Z

If you want to break this into 3 PRs:

design doc
secrets type definition
volume plugin and secrets watching impl.

I won't complain. 😄

pmorie · 2015-02-18T01:03:15Z

@erictune thanks for the feedback. I will split out the PRs as you
suggested.
On Tue, Feb 17, 2015 at 7:52 PM Eric Tune notifications@github.com wrote:

If you want to break this into 3 PRs:

design doc

secrets type definition

volume plugin and secrets watching impl.

I won't complain. [image: 😄]

—
Reply to this email directly or view it on GitHub
#4126 (comment)
.

pmorie · 2015-02-18T01:31:44Z

@erictune I have turned this into the PR for the proposal, separate PRs coming for type definition (needs a little cleanup and final review) and the volume plugin (where we can work through tmpfs)

Secrets proposal

thockin · 2015-02-20T04:56:27Z

I just wanted to say that I finally read this whole thing and (aside from some nits which I will address as a PR or just ignore) I found it to be incredibly well written and thorough. This is a model design doc.

erictune · 2015-02-20T19:41:28Z

+1

On Thu, Feb 19, 2015 at 8:56 PM, Tim Hockin notifications@github.com
wrote:

I just wanted to say that I finally read this whole thing and (aside from
some nits which I will address as a PR or just ignore) I found it to be
incredibly well written and thorough. This is a model design doc.

—
Reply to this email directly or view it on GitHub
#4126 (comment)
.

Secrets proposal

smarterclayton mentioned this pull request Feb 4, 2015

Kubernetes pod and namespace security model #4029

Merged

markturansky reviewed Feb 6, 2015
View reviewed changes

j3ffml assigned smarterclayton Feb 6, 2015

pmorie force-pushed the secrets_proposal branch from fa7b6bb to 87b717a Compare February 8, 2015 16:25

pmorie force-pushed the secrets_proposal branch 2 times, most recently from 375d4bf to 3d37204 Compare February 11, 2015 04:59

pmorie force-pushed the secrets_proposal branch from 3d37204 to 45244d9 Compare February 11, 2015 06:19

erictune reviewed Feb 11, 2015
View reviewed changes

EricMountain-1A reviewed Feb 17, 2015
View reviewed changes

pmorie force-pushed the secrets_proposal branch 3 times, most recently from 27d8018 to e26050a Compare February 17, 2015 21:40

erictune reviewed Feb 18, 2015
View reviewed changes

pmorie force-pushed the secrets_proposal branch 2 times, most recently from 361b977 to c80ec0a Compare February 18, 2015 01:28

Secrets proposal

ea18e66

pmorie force-pushed the secrets_proposal branch from c80ec0a to ea18e66 Compare February 18, 2015 01:30

pmorie changed the title ~~WIP: Secrets proposal / prototype~~ Secrets proposal Feb 18, 2015

This was referenced Feb 18, 2015

Secret API resource #4514

Merged

Secret volume plugin iteration 1 #4515

Merged

erictune added a commit that referenced this pull request Feb 18, 2015

Merge pull request #4126 from pmorie/secrets_proposal

e045b19

Secrets proposal

erictune merged commit e045b19 into kubernetes:master Feb 18, 2015

pmorie mentioned this pull request Feb 19, 2015

Kubelet secret volume plugin should store data in tmpfs #4602

Closed

pmorie mentioned this pull request Feb 22, 2015

Expose secrets to containers in environment variables #4710

Closed

RichieEscarez mentioned this pull request Jul 6, 2015

Make docs links be relative so we can version them #10681

Merged

xingzhou pushed a commit to xingzhou/kubernetes that referenced this pull request Dec 15, 2016

Merge pull request kubernetes#4126 from pmorie/secrets_proposal

77211be

Secrets proposal

iholder101 mentioned this pull request Jul 19, 2023

Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) #118865

Merged

Secrets proposal #4126

Secrets proposal #4126

Conversation

pmorie commented Feb 4, 2015

roberthbailey commented Feb 4, 2015

smarterclayton commented Feb 4, 2015

markturansky commented Feb 4, 2015

pmorie commented Feb 5, 2015

erictune commented Feb 5, 2015

markturansky commented Feb 5, 2015

markturansky commented Feb 5, 2015

pmorie commented Feb 5, 2015

markturansky commented Feb 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markturansky commented Feb 7, 2015

smarterclayton commented Feb 7, 2015

markturansky commented Feb 7, 2015

smarterclayton commented Feb 7, 2015

pmorie commented Feb 8, 2015

smarterclayton commented Feb 8, 2015

markturansky commented Feb 8, 2015

smarterclayton commented Feb 10, 2015

markturansky commented Feb 10, 2015

pmorie commented Feb 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmorie commented Feb 17, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erictune commented Feb 18, 2015

erictune commented Feb 18, 2015

pmorie commented Feb 18, 2015

pmorie commented Feb 18, 2015

thockin commented Feb 20, 2015

erictune commented Feb 20, 2015