Relevant to #4672

@pmorie

@bgrant0607

I'm really hoping to avoid having the kubelet be dynamically updating magic
volumes (run your own side-car), but I am not sure we can escape it with
secrets since there is info there that is meaningful to the kubelet itself..

On Fri, Feb 20, 2015 at 9:43 AM, Eric Tune notifications@github.com wrote:

@bgrant0607 https://github.com/bgrant0607

Reply to this email directly or view it on GitHub
#4673 (comment)
.

I think we should support dynamic updates. That makes the feature vastly more useful and reduces the complexity for clients tremendously. Immutable secret objects would induce cascading complexity: automatic name generation, automatic garbage collection of secrets not in use, etc. Plus, I doubt we could prevent users from updating the volume source to point to a new secret, which seems to have about the same complexity. In general, making things not updatable greatly complicates our user-facing configuration solution. We could potentially restart/recreate containers upon these kinds of updates for now, if that would simplify the implementation.

Additionally, we're going to want the same behavior for a dynamic config solution for our own components.

Also, I vote for atomic update. A lifecycle event can notify the containers that they should reopen the file(s).

Dynamic updates can be achieved by running a secret-syncer sidecar. I'm
just trying to convince myself that this is not a viable answer.

As for atomic updates, easier said than done. The only atomic op is
rename() and that is singular AND only works for directories iff the
destination does not already exist.

We could do something like put secrets into a subdir of the volume and then
flip a symlink with rename.

e.g.

mkdir /path/to/volume/.update_in_progress
write secrets to /path/to/volume/.update_in_progress
mv /path/to/volume/.update_in_progress /path/to/volume/new_timestamp
ln -s new_timestamp /path/to/volume/.updated_link
rename /path/to/volume/.updated_link /path/to/volume/current

It's ugly and we still have to deal with cleanup, but it's the only way I
know.

On Fri, Feb 20, 2015 at 10:48 PM, Brian Grant notifications@github.com
wrote:

Also, I vote for atomic update. A lifecycle event can notify the
containers that they should reopen the file(s).

Reply to this email directly or view it on GitHub
#4673 (comment)
.

@erictune @thockin @bgrant0607 I am torn on this one personally. One thing I want to point out while I gather my thoughts is that I think we have a similar problem in this use-case to other ones that have been mentioned:

• Expose labels to containers (Pass pod labels to containers and to lifecycle hooks #560)
• Dynamic configuration (Support dynamic configuration distribution #1553)
• Exposing secrets as env vars (Expose secrets to containers in environment variables #4710)

I now think we probably don't need to prioritize automated rotation at this time.

Most pods are going to have a way that they can be updated to have a new image or new flags. Updating the secret can use that same workflow, assuming secret changes are infrequent.

There are two cases where secrets might change frequently, but I don't think these are that important right now:

First, short lived tokens. We've talked about having a way to hold something like an OAuth Refresh Token in the apiserver, and then have it generate access tokens periodically, and "rotate" them for clients. While this has the nice property of protecting the long lived secret, clients are probably not going to expect to have the access token changing underneath them, and it has the drawback of giving the client the access token even when it doesn't need it (if it has time-varying behavior).

Second would be a server that stores multiple changing secrets, like username/password combinations for a website. This case doesn't match the current secrets implementation: secret modifications made by the pod won't be remembered.

TL;DR
I don't think we should work on secret rotation right now.
And we should make secrets immutable. I'll file a narrower issue for that.

I'm fine closing this for now - it's a much larger design issue and not priortiy