Can you link to the discussion on config map refresh?

We've gone over secret refreshing several times, and every time @erictune and I justified it not being refreshed. Will want a clear argument for it being required or expected.

I was also surprised when I heard config map can refresh. Does it update atomically?

Yes it updates atomically (well, it will - downward API does, or at least I
am pretty sure it does).

On Wed, Dec 9, 2015 at 4:05 PM, Eric Tune notifications@github.com wrote:

I was also surprised when I heard config map can refresh. Does it update
atomically?

—
Reply to this email directly or view it on GitHub
#18372 (comment)
.

Downward API updates atomically. We talked about config volume
refreshing atomically too, in the proposal. I will comb through the
horrid github UI to find some links if people want.

On Thu, Dec 10, 2015 at 2:05 AM, Tim Hockin notifications@github.com
wrote:

Yes it updates atomically (well, it will - downward API does, or at least I
am pretty sure it does).

On Wed, Dec 9, 2015 at 4:05 PM, Eric Tune notifications@github.com
wrote:

I was also surprised when I heard config map can refresh. Does it update
atomically?

—
Reply to this email directly or view it on GitHub
<
#18372 (comment)

.

—
Reply to this email directly or view it on GitHub
#18372 (comment)
.

I think there are 2 patterns people will use with ConfigMap:

1. Update the ConfigMap and expect it to immediately propagate to all instances. They may watch their configurations using using inotify, or expect a HUP, or just restart. This is useful for applications where all replicas need consistent configuration. I'd guess etcd and Zookeeper are in this category.
2. Create a new ConfigMap, update the pod template to reference it, and roll it out via rolling update. This is useful when replicas don't need identical configuration and one is worried about pushing bad configs, which is a common source of failure.

Updating ConfigMap and not propagating the change is just confusing. Expecting users to kill pods in order to implicitly pick up changes lacks transparency and predictability, which is why we moved away from that approach in rolling update.

Other than "it was simpler to implement", what's the rationale for not propagating Secret updates?

I think it was @erictune who argued that updating Secrets was semantically
weird for users. My memory fails me, though.

On Thu, Jan 7, 2016 at 7:58 PM, Brian Grant notifications@github.com
wrote:

I think there are 2 patterns people will use with ConfigMap:

Update the ConfigMap and expect it to immediately propagate to all
instances. They may watch their configurations using using inotify, or
expect a HUP, or just restart. This is useful for applications where all
replicas need consistent configuration. I'd guess etcd and Zookeeper are in
this category.
2.

Create a new ConfigMap, update the pod template to reference it, and
roll it out via rolling update. This is useful when replicas don't need
identical configuration and one is worried about pushing bad configs, which
is a common source of failure.

Updating ConfigMap and not propagating the change is just confusing.
Expecting users to kill pods in order to implicitly pick up changes lacks
transparency and predictability, which is why we moved away from that
approach in rolling update.

Other than "it was simpler to implement", what's the rationale for not
propagating Secret updates?

—
Reply to this email directly or view it on GitHub
#18372 (comment)
.

I think the idea that I have to stop my application (aka delete the pod) to update a secret or config is weird... I should be able to update the secret/config and send my app a SIGHUP.

I'm all for automatic update of the files in existing pods...

Eric and I both discussed this at the last face to face (last year) and
agreed that secret changing led to all sorts of weird scenarios that didn't
need anything other than deployment to solve.

On Tue, Jan 12, 2016 at 12:25 PM, Eric Paris notifications@github.com
wrote:

I think the idea that I have to stop my application (aka delete the pod)
to update a secret or config is weird... I should be able to update the
secret/config and send my app a SIGHUP.

I'm all for automatic update of the files in existing pods...

—
Reply to this email directly or view it on GitHub
#18372 (comment)
.

I spent some time today combing through old github comments, attempting to unearth our prior discussion, and was unsuccessful. Does anyone have a reason not to do this? I do seem to remember as @thockin does that @erictune had some concerns about updating secrets but wasn't able to find them in the github issues I looked through. Eric, do you happen to remember any of those, or even better, have links to github comments you expressed those reservations in memorized?

As an aside, it truly, truly sucks trying to find old comments via the github web UI.

#4673, #4950 and #4672 seem related.

Some issue I can think of:

• not all clients will know to read the new secret file without restarting. For example our own client library doesn't know how to do this. See Key/Cert rotation for Kubernetes clients. #4672. Reasoning about whether the change was picked up is hard if you don't restart everything. Users may know it is safe for individual apps, but any kind of automatic rotation system won't know this.
• if clients keep the old file open, you can't free up the inode, and so use extra memory.
• I don't think we knew how to update the files in the tmpfs in a way that appeared atomic to the program in the container. something about bind mounts. see Secrets rotation and Secrets semantics #4673. Did we solve this for configMap? I recall this being a hard problem.

On Sun, Jan 31, 2016 at 2:52 PM, Eric Tune notifications@github.com wrote:

• not all clients will know to read the new secret file without restarting. For example our own client library doesn't know how to do this. See Key/Cert rotation for Kubernetes clients. #4672. Reasoning about whether the change was picked up is hard if you don't restart everything. Users may know it is safe for individual apps, but any kind of automatic rotation system won't know this.

True. OTOH, live updates seems like something you should try before
you deploy, and if it doesn't work for you, don't use it.

if clients keep the old file open, you can't free up the inode, and so use extra memory.

IMO this is a relatively minor point

I don't think we knew how to update the files in the tmpfs in a way that appeared atomic to the program in the container. something about bind mounts. see #4673. Did we solve this for configMap? I recall this being a hard problem.

We understand how to update files within a mounted directory.
Atomically updating files individually mount files is the unsolved
problem (well, solved but uuuuuugly)

I chatted with @erictune yesterday about this issue. Where we settled is: we want to make ConfigMap and Secret as close to one another as possible, and people want the features that the atomic writer offers for use with secrets, so we should:

1. Make Secrets updateable
2. Add the ability to do path projections into the volume ala downward API and ConfigMap
3. Refactor the secret volume plugin to use the atomic writer

We talked about adding a way to SIGHUP containers that mount the volume when content is updated. I think users want this but I don't know that we settled on whether to do it right now; will file a separate issue for that. I'll note here that Eric brought up that their may be races when signalling containers, and we should detect containers referencing old versions of files.

Sadly, since we can't yet share PID namespaces, signals will have to come
from outside (blech) rather than a sidecar.

On Sun, Apr 3, 2016 at 9:40 PM, Paul Morie notifications@github.com wrote:

I chatted with @erinctune yesterday about this issue. Where we settled is:
we want to make ConfigMap and Secret as close to one another as possible,
and people want the features that the atomic writer offers for use with
secrets, so we should:

1. Make Secrets updateable
2. Add the ability to do path projections into the volume ala downward
   API and ConfigMap
3. Refactor the secret volume plugin to use the atomic writer

We talked about adding a way to SIGHUP containers that mount the volume
when content is updated. I think users want this but I don't know that we
settled on whether to do it right now; will file a separate issue for that.
I'll note here that Eric brought up that their may be races when signalling
containers, and we should detect containers referencing old versions of
files.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#18372 (comment)

💩

Re: picking up new config:

The user-centric question for configmap updates (and secrets now that we are going to allow updates) is I updated my configMap/Secret... Now when should I expect to see my app use the new files?

If we think the system should be able to answer this for the user, then it would need to know the semantics of the application in the container. We thought there were 3 styles of applications:

1. apps that need to restart to read config.
2. apps that reread config after a signal (HUP being the classic example)
3. apps that poll or inotify to detect file changes and reload

Users of secrets and configmap need to be aware which type of app they have, so they can know what the steps are to complete a config update / secret rotation for their app.

I wonder if we should define something like:

type ConfigPushAction string

const (
        // Restart means that the app requires restart to pick up changes to a configuration file.
        ConfigPushActionRestart ConfigPushAction = "Restart"
        // Hup means that the app requires a sighup to pickup changes to a configuration file.
        ConfigPushActionHup ConfigPushAction = "Hup"
        // None means that the app can detect new config files and so requires no action after config is pushed.  
        ConfigPushActionNone ConfigPushAction = "None"
)
...
type Container struct {
...
   ConfigPushAction ConfigPushAction `omitempty, name:configPushAction`
...
}

It would get tricky though when you start to get into which pid to signal (maybe better to use the ExecProbe model instead, or for special cases). It also gets tricky if different files have different semantics.

But, if you get it right, then you can automate more of the update process.

Do I see it correctly that there's currently no way to check whether/when a configmap volume update happened?
It's great that the configmap volumes resfresh. But it seems difficult to make reliable use of it right now aside from manually and periodically sending SIGHUP as the refreshing takes some time to propagate.

@erictune's suggestion seems very reasonable from my perspective. How's the general sentiment for implementing it?

I'd be fine with adding a notifier concept - I bet it comes up in other
contexts eventually, so we should describe it generically. something like

type PodNotifier struct {
    Type PodNotifierType
    Signal *SignalPodNotifier
    HTTPPost *HTTPPostPodNotifier
}

// Send a signal to the root process of the container.
type SignalPodNotifier struct {
    ContainerName string
    Name SignalName
}

// POST a context-dependent body to signal the app.
type HTTPPostPodNotifier struct {
  Port intstr.IntOrString
  Path string
  Scheme URIScheme
  HTTPHeaders []HTTPHeader
 }

On Tue, Apr 19, 2016 at 4:57 AM, Fabian Reinartz notifications@github.com
wrote:

Do I see it correctly that there's currently no way to check whether/when
a configmap volume update happened?
It's great that the configmap volumes resfresh. But it seems difficult to
make reliable use of it right now aside from manually and periodically
sending SIGHUP as the refreshing takes some time to propagate.

@erictune https://github.com/erictune's suggestion seems very
reasonable from my perspective. How's the general sentiment for
implementing it?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#18372 (comment)

@fabxc

Do I see it correctly that there's currently no way to check whether/when a configmap volume update happened?

You can watch the volume's ..data symlink with inotify or fanotify

I was thinking of something on the Kubernetes side.
It's not a too scalable approach to make all apps that reload via SIGHUP implement file watches.

For Prometheus at least, it was a very intentional choice to not us inotify for the configuration due to a variety of issues encountered that made it an unfeasible cross-platform solution.

@fabxc you mentioned in your post

It's great that the configmap volumes resfresh.

I could not find a way to trigger it via cli (kubectl), does it exist only as API? If you know how to use configMap update/refresh with cli, appreciate if you could share.

@sbezverk i haven't tried it, but it is referenced here: https://github.com/kubernetes/kubernetes/blob/master/docs/design/configmap.md

If it doesn't refresh automatically, is there a way to trigger a volume refresh manually without having to kill the pod? Use case: updating nginx config and sending it a HUP signal

@sgrimee not yet, it has been requested though, will link later.

err, I answered the wrong question. It does refresh automatically, but it's not instantaneous. The volume is eventually consistent, just like downward API volume.

@pmorie this is done now, yes? Or is there more to do?

nope, it's done.

On Friday, June 3, 2016, Tim Hockin notifications@github.com wrote:

@pmorie https://github.com/pmorie this is done now, yes? Or is there
more to do?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#18372 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAWXmE5M2TTlq7PmLtBzMdujPWFkr1pRks5qIIAWgaJpZM4GxPNV
.

Closed by #25285

Can this refresh be enabled/disabled? It's causing me issues when renaming the files in a mounted secret and changing permissions, see #28898

I've read through this and several related issues, not to mention any documentation I can find. It seems that config reloads for pods that use HUP signals is implemented; I just don't see how to implement it. Perhaps I'm just obtuse, but it would make sense to me if there was a way to define this in a configmap, deployment, service, or container spec so that when I update a configmap that a pod is using that the pod updates the config file from the API and then sends the correct HUP signal.

Did I miss something obvious? Would really appreciate someone pointing me in the right direction. My use case is NGINX as a load balancer for InfluxDB/relay. I'm attempting an HA configuration and will need nodes to be able to update the NGINX configmap as they enter various stages of the recovery process; and I need to do this without dropping connections (i.e. destroying and re-creating the NGINX pod).

Again - any help appreciated.