Continuation of the `docker secret` storage feature #6697

vbatts · 2014-06-26T14:56:44Z

Starting on a new PR to accommodate alex being on holiday

SvenDowideit · 2014-06-27T03:20:33Z

docs/sources/reference/commandline/cli.md

+
+This is an example of using the secrets database in building an image that will
+have buildtime access to a sensitive file, that will not be committed in the
+final image.


This example does not show how one would use the secrets file, nor how to clean up afterwards - for example - are you expecting users to softlink the secret file to its expected location?

How do you then manage the hidden dependency on a file that is then gone? will a subsequent docker run give the user a sensible error message (before the container starts?)

it seems useful to me to add a local secret file mapped into my /etc/apt/sources.d so that I get my local debian repo - but doing so will either leave a dangling sofltlink, or some kind of confusion.

I'm starting to think that secret is a wrong name for this.

Dangling symlinks, that is what we are expecting. Which matches expectations. If the secret is no longer present, then the dangling link will fail.

vbatts · 2014-07-15T15:24:52Z

@SvenDowideit I am not opposed to splitting the global secrets out to a separate PR, but there is a definite use-case for global secrets, rather than having to explicitily grant secrets for every run and build.

As for the name "secrets", perhaps there could be other names, though that is the essence of the conveyance. Something like token, keys, and certificates.

thaJeztah · 2014-07-15T21:13:47Z

...perhaps there could be other names

Hm, just some terms that come in mind, for inspiration;

vault
depot (missed opportunity - this should have been de name for docker hub!)
protected storage / secured storage
locker
keychain (although this implies that it is limited to certificates storage)

SvenDowideit · 2014-07-15T22:51:25Z

thing is - the functionality can inject pretty much anything - you're talking secrets, I'm thinking I can inject my data just as easily.

vbatts · 2014-07-16T00:06:04Z

@thaJeztah agreed. Something like 'vault' would be closer, to what is happening.

@SvenDowideit would that semantic be more inline with the functionality?

SvenDowideit · 2014-07-16T03:41:59Z

it seems less shifty :) - and if I can tell a container run that I want to put vault item X at location Y in the container, then we're being blatantly obvious about it.

ie - when I run a debian container, i might have a vault item local-apt-cache.conf that I will inject by doing something like docker run --inject local-apt-cache.conf:/var/apt/sources.list debian bash

or similarly at build time.

and in writing that rather obvious documentation - can you please remind me why you're not using --volume bind mounts, and talking about mitigating the concerns there are about adding build time volumes wrt portability?

timthelion · 2014-07-16T06:41:59Z

@SvenDowideit the whole point of naming this "secrets" was to have "build time volumes with a name that doesn't suggest that they should be used for anything but redhat's product key hackery". It's not supposed to be named something friendly. Its specifically intended to be named something that doesn't sound general purpose-y.

thaJeztah · 2014-07-16T06:53:03Z

@timthelion if it's only to be used for "redhat's product key hackery", and not to be used for other purposes, should it be implemented at all? should it be documented?

Playing devil's advocate here :)

timthelion · 2014-07-16T11:16:23Z

@thaJeztah There is a non-redhat @shykes approved use for this feature; signing of builds with a private GPG key. You don't want the private key to stay in the docker image once you're done ;)

SvenDowideit · 2014-07-17T00:51:34Z

please refer to my example of a non-secret use that this feature would be useful for.

However - the image portability risks do need to be addressed.

One possible resolution is that a secret should replace an existing default file - so that the image works (to whatever degree is reasonable) with the local secret - my apt sources file is a good eg, but also shows that there are flaws (what if I want to add a new file to the /etc/apt/sources.d/ dir instead.

with my support hat on - good luck in keeping this feature even a semi-secret - why not design it properly instead?

thaJeztah · 2014-07-17T10:57:07Z

@SvenDowideit @timthelion Basically that's what I was pointing at; If it's for internal use only, don't document it and don't make it accessible via the API.

However, since it's being developed for external use (red hat, signing builds) - although for very special cases, it must be properly documented, maybe just to point out what it is meant to be used for and what not. Maybe even to warn people that it is an unsupported 'feature'.

Hiding things doesn't really work in open-source projects, does it? :)

Signing off now, because this is probably heading in the wrong direction, distracting from the main purpose of this issue :)

vbatts · 2014-08-15T19:40:03Z

rebased

thaJeztah · 2014-08-15T19:44:43Z

@vbatts any discussion still going on about the naming of this? Still in favour of vault personallly.

vbatts · 2014-08-15T20:05:13Z

@thaJeztah there hasn't been much conversation on the naming. I think the bigger hang-up is on the global secrets, rather than just explicit grants.

thaJeztah · 2014-08-15T20:11:31Z

@vbatts thx. Just wanted to give the naming a bit of attention before it was merged and I'm too late to mention it 😸

vbatts · 2014-10-02T19:31:16Z

meeting notes https://botbot.me/freenode/docker-dev/msg/22757310/

defunctzombie · 2014-10-05T22:06:12Z

Would the local secrets file be something that lives in the repo alongside the Dockerfile? I think this would be useful and allow for reproducible builds by having the secrets file be under source control. The secrets file could be encrypted when on disk (like ansible vault. Then for docker build you would be able to provide the passphrase to unlock the secrets file (and the file is already available since it was sent with the build context).

vbatts · 2014-10-06T15:29:59Z

@defunctzombie not. That is both the benefit and drawback of the concept of secrets. That they are managed independently and subject to change.

The secrets store keeps track of a set of named secrets, such as API keys or ssh keys that you can later access from containers. There are two types of secrets, the host based ones, which are stored in /etc/docker/secrets, and are automatically applied to all containers, and user secrets which you can add via "docker secret add <name> <file>" and which can be granted permission to by name. You can also list secrets with "docker secret list" and remove them with "docker secret rm". Add --grant-secret support to docker build This is useful if you want to use secrets during docker build which will not be recorded in the final image. API docs: add the new secret based requests docs: Add cli docs for the secret store cli: Allow multi-level command like docker foo list, and docker foo add If you implent CmdFoo with an extra boolean first arg, and CmdFooList and CmdFooAdd as usual the code will make this just work. vbatts: * adding and clarifying docs * fix the change in error handling * rebasing * removed global/host secrets Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson) Signed-off-by: Vincent Batts <vbatts@redhat.com>

vbatts · 2014-10-08T18:11:21Z

alrighty.
much rebasing. very secrets. wow.

also I squashed the commits down because it was such an overhaul to rebase and the sub command feature has since been implemented another way.

There are still a few issues with the implementation that I'll like to address, but notably this PR now does not include the global/host secrets, only added secrets that have to be explicitly granted.

timthelion · 2014-10-09T06:52:02Z

daemon/container.go

@@ -775,6 +800,10 @@ func (container *Container) jsonPath() (string, error) {
 	return container.getRootResourcePath("config.json")
 }

+func (container *Container) secretsPath() (string, error) {
+	return container.getRootResourcePath("secrets")


I sugest you place the secrets in container.getRootResourcePath("secrets/secret-files") this way it will be easy to add a container.getRootResourcePath("secrets/secret-properties.json") file in the future.

Good thought. That would make it easier for a next step of handling ACLs of
who added the secret and what is the permissions on it.
On Oct 9, 2014 4:06 AM, "Timothy Hobbs" notifications@github.com wrote:

In daemon/container.go:

@@ -775,6 +800,10 @@ func (container *Container) jsonPath() (string, error) {
return container.getRootResourcePath("config.json")
}

+func (container *Container) secretsPath() (string, error) {

return container.getRootResourcePath("secrets")

I sugest you place the secrets in
container.getRootResourcePath("secrets/secret-files") this way it will be
easy to add a
container.getRootResourcePath("secrets/secret-properties.json") file in
the future.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/6697/files#r18629350.

shykes · 2014-11-04T19:35:07Z

After design discussion:

Nobody can agree on anything. Clearly the proposal encompasses too many use cases (and the data is scarce on actual usage)
Let's split up the use cases, and focus on each separately.
Specifically there is 1) injecting host-specific credentials aka the "rhel key use case", and 2) injecting developer-specific credentials at build aka the "developer ssh keys use case".

I will welcome specific docs-first proposals for solving each of these use cases, separately.

Sorry that this is taking so long.

defunctzombie · 2014-11-09T00:22:41Z

shameless plug Since this issue is now closed, I figured I would share my stop-gap solution docket which I currently use to build images that require sensitive information. Docket is built on top of docker build and does some tricks to ensure that your sensitive information is not present in the final image or image history.

The project is still relatively new and experimental, but I welcome contributions and feedback from anyone that wishes to try it!

razic · 2015-04-08T21:33:12Z

@shykes Before I go and create a proposal for the "dev ssh key for builds" use case, I just want to make sure there aren't any currently open.

This one is closed and #10310 doesn't address an individual use case like you asked for on Nov 4th.

I couldn't find any other related proposals but just wanted to make sure. Can you confirm?

ilan-schemoul · 2017-01-05T21:40:36Z

Damn we're in 2017 and after reading tons of PR and articles the only thing I learned is that Docker has this problem for a long time even tough a lot of PR has been proposed. Weirdo.

timthelion · 2017-01-05T21:49:24Z

The trouble that happened was that during one of the regular IRC meetings a design was proposed by three people and agreed upon by @shykes (I don't even recal the exact design) a seccond design was proposed by like two people and implemented by Alex Larson. And the two designs were very different and @shykes explicitly said during the meeting that he didn't like the one that Alex proposed. And so it lead to this weird situation of having an implemented PR that was against the wishes of the BDFL and no PR that was according to his wishes. Now I think that shykes is quite busy right now, and so maybe his oppinion matters less and this PR can go through. Or someone can go look up those old IRC conversations and try to implement the "correct" design.

thaJeztah · 2017-01-06T01:17:47Z

@NitroBAY @timthelion have a look at #27794 (which will be in docker 1.13), and #28079 (which probably will be continued on for docker 1.14)

ilan-schemoul · 2017-01-06T15:25:17Z

Okay thanks, I may be suprised because I'm used to such huge projects, it's maybe normal that implement a feature is that complex in that kind of projects.

vbatts mentioned this pull request Jun 26, 2014

Add Secret store #6075

Closed

vbatts added the /project/doc label Jun 26, 2014

erikh added Trust and removed Orchestration labels Jun 26, 2014

SvenDowideit reviewed Jun 27, 2014
View reviewed changes

vieux added this to the 1.1.1 milestone Jul 2, 2014

timthelion mentioned this pull request Jul 19, 2014

Dockerfiles should have a way to perform multiple build actions in one commit #2439

Closed

vieux modified the milestones: 1.1.1, 1.3.0 Aug 22, 2014

crosbymichael removed this from the 1.3.0 milestone Sep 23, 2014

cpuguy83 mentioned this pull request Sep 24, 2014

Proposal: Dockerfile add BIND_CONTEXT #3056

Closed

vbatts force-pushed the alexlarsson-secrets branch from 36ca0b0 to e9b3bb9 Compare October 8, 2014 18:07

timthelion reviewed Oct 9, 2014
View reviewed changes

kanzure mentioned this pull request Oct 17, 2014

Forward ssh key agent into container #6396

Open

crosbymichael self-assigned this Oct 17, 2014

clifton mentioned this pull request Oct 17, 2014

Allow forwarding SSH agent in fig docker/compose#551

Closed

shykes closed this Nov 4, 2014

djmaze mentioned this pull request Nov 4, 2014

Proposal: sensitive data during build #8958

Closed

cyphar mentioned this pull request Jan 23, 2015

Proposal: The Docker Vault #10310

Closed

thaJeztah mentioned this pull request May 26, 2015

Secrets: write-up best practices, do's and don'ts, roadmap #13490

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Continuation of the `docker secret` storage feature #6697

Continuation of the `docker secret` storage feature #6697

vbatts commented Jun 26, 2014

SvenDowideit Jun 27, 2014

SvenDowideit Jun 27, 2014

vbatts Jul 15, 2014

vbatts commented Jul 15, 2014

thaJeztah commented Jul 15, 2014

SvenDowideit commented Jul 15, 2014

vbatts commented Jul 16, 2014

SvenDowideit commented Jul 16, 2014

timthelion commented Jul 16, 2014

thaJeztah commented Jul 16, 2014

timthelion commented Jul 16, 2014

SvenDowideit commented Jul 17, 2014

thaJeztah commented Jul 17, 2014

vbatts commented Aug 15, 2014

thaJeztah commented Aug 15, 2014

vbatts commented Aug 15, 2014

thaJeztah commented Aug 15, 2014

vbatts commented Oct 2, 2014

defunctzombie commented Oct 5, 2014

vbatts commented Oct 6, 2014

vbatts commented Oct 8, 2014

timthelion Oct 9, 2014

vbatts Oct 9, 2014

shykes commented Nov 4, 2014

defunctzombie commented Nov 9, 2014

razic commented Apr 8, 2015

ilan-schemoul commented Jan 5, 2017

timthelion commented Jan 5, 2017 via email

thaJeztah commented Jan 6, 2017

ilan-schemoul commented Jan 6, 2017

Continuation of the docker secret storage feature #6697

Continuation of the docker secret storage feature #6697

Conversation

vbatts commented Jun 26, 2014

SvenDowideit Jun 27, 2014

Choose a reason for hiding this comment

SvenDowideit Jun 27, 2014

Choose a reason for hiding this comment

vbatts Jul 15, 2014

Choose a reason for hiding this comment

vbatts commented Jul 15, 2014

thaJeztah commented Jul 15, 2014

SvenDowideit commented Jul 15, 2014

vbatts commented Jul 16, 2014

SvenDowideit commented Jul 16, 2014

timthelion commented Jul 16, 2014

thaJeztah commented Jul 16, 2014

timthelion commented Jul 16, 2014

SvenDowideit commented Jul 17, 2014

thaJeztah commented Jul 17, 2014

vbatts commented Aug 15, 2014

thaJeztah commented Aug 15, 2014

vbatts commented Aug 15, 2014

thaJeztah commented Aug 15, 2014

vbatts commented Oct 2, 2014

defunctzombie commented Oct 5, 2014

vbatts commented Oct 6, 2014

vbatts commented Oct 8, 2014

timthelion Oct 9, 2014

Choose a reason for hiding this comment

vbatts Oct 9, 2014

Choose a reason for hiding this comment

shykes commented Nov 4, 2014

defunctzombie commented Nov 9, 2014

razic commented Apr 8, 2015

ilan-schemoul commented Jan 5, 2017

timthelion commented Jan 5, 2017 via email

thaJeztah commented Jan 6, 2017

ilan-schemoul commented Jan 6, 2017

Continuation of the `docker secret` storage feature #6697

Continuation of the `docker secret` storage feature #6697