Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distroless container doesn't have the kill cmd #199

Closed
mkmik opened this issue Jul 25, 2019 · 12 comments
Closed

Distroless container doesn't have the kill cmd #199

mkmik opened this issue Jul 25, 2019 · 12 comments

Comments

@mkmik
Copy link
Collaborator

mkmik commented Jul 25, 2019

The README instructs to send a USR1 signal to trigger a key rotation, but since #174 the docker image contains only controller static binary and the essential files (in particular there is no shell)

$ kubectl exec -it <controller pod> -- kill -SIGUSR1 1
OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"kill\": executable file not found in $PATH": unknown

We could either use the "debug" distroless image flavour (which contains busybox) or figure out (and document) another way to trigger the key rotation)

@mkmik mkmik added the bug label Jul 25, 2019
@jjo
Copy link

jjo commented Jul 25, 2019

Maybe implement it in the controller itself (or a helper CLI as e.g.
controller-ctl), e.g. controller ctl reload.

That way we can currently implement it ~crudely via kill -USR1 1, i.e.
"fixing" the CLI interface as ctl reload while leaving space for improvement
should we need it (e.g. adding other verbs than reload, change controlling
interface from signalling to unix socket, etc).

@mkmik
Copy link
Collaborator Author

mkmik commented Jul 25, 2019

Nice idea. Some practicalities: how would controller ctl reload find the controller pid? Assuming it's pid 1 seems a bit brittle.
Should the controller create a pid file? or a unix domain socket and accept commands through it?

@jjo
Copy link

jjo commented Jul 26, 2019

Nice idea. Some practicalities: how would controller ctl reload find the controller pid? Assuming it's pid 1 seems a bit brittle.

Yep, it is - just as we currently have documented it :tada+troll:

Couple thoughts:

  • this should be interim~ish only really, as exec -it <pod> is not a
    sustainable/want-able control interface really (more below)
  • implementation wise, we could implement it (~ controller ctl reload)
    by doing the equivalent of:
[ $(readlink /proc/1/exe) = $0 ] && kill -USR1 1 || echo "ERROR: pid 1 is not the controller"

i.e. check that pid==1 is running the same executable (path) as me,
to de-brittle that kill it a bit ;)

Should the controller create a pid file? or a unix domain socket and accept commands through it?

I wouldn't move forward that Kube-unfriendly path IMO, we should rather
think on e.g. watching a configMap for changes for a
rotation_time_req: <epoch> field which would need a
rotation_time_last: <epoch> written by the controller for state.

@mkmik
Copy link
Collaborator Author

mkmik commented Jul 26, 2019

yeah, I'd love a declarative approach.

what if we just had a flag/env:

flag.String("rotate-if-older-than", 0, "rotate the key if there is no secret sealedsecrets.bitnami.com/sealed-secrets-key=active more recent that the given timestamp")

(the we can feed it via a CM, because it's easier to fiddle with kustomize than args I guess)

Can you think of a better name than rotate-if-older-than?

rotation_time_last: written by the controller for state.

do we need that?

We should emit an Event when rotation happens +
the user can see the side effect (kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key=active)

@mkmik
Copy link
Collaborator Author

mkmik commented Jul 26, 2019

BTW, about: flag.Duration("rotate-period", 0, "New key generation period (automatic rotation disabled if 0)")

// TODO(mkm): implement rotation cadence based on resource times rather than just an in-process timer.

Once we have this we can use the same machinery to check whether to rotate based on absolute cut-off time or period (cut-off time relative to the creation time of the latest active secret)

@mkmik
Copy link
Collaborator Author

mkmik commented Jul 26, 2019

if we do that, should we just kill the SIGUSR1 feature (no pun intended)

@jjo
Copy link

jjo commented Jul 26, 2019

yeah, I'd love a declarative approach.

what if we just had a flag/env:

flag.String("rotate-if-older-than", 0, "rotate the key if there is no secret sealedsecrets.bitnami.com/sealed-secrets-key=active more recent that the given timestamp")

(the we can feed it via a CM, because it's easier to fiddle with kustomize than args I guess)

Can you think of a better name than rotate-if-older-than?

maybe using the well-known TTL acronym for something like
private-key-ttl ? (or maybe add rotation somewhere to make
the action explicit)

rotation_time_last: written by the controller for state.

do we need that?

Missed the fact that we can peek on the priv key timestamp,
thus indeed not needed.

We should emit an Event when rotation happens +
the user can see the side effect (kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key=active)

+1

@mkmik
Copy link
Collaborator Author

mkmik commented Jul 26, 2019

maybe using the well-known TTL acronym for something like
private-key-ttl ? (or maybe add rotation somewhere to make
the action explicit)

hmm; perhaps it's just me, but isn't TTL more often used to express relative times (basically the equivalent of what we currently call "period"), i.e. for how long are new private keys valid before we replace them with a new one (which would in turn also live for TTL seconds before getting in turn replaced by a new one and so on)?

@jjo
Copy link

jjo commented Jul 26, 2019

maybe using the well-known TTL acronym for something like
private-key-ttl ? (or maybe add rotation somewhere to make
the action explicit)

hmm; perhaps it's just me, but isn't TTL more often used to express relative times (basically the equivalent of what we currently call "period"), i.e. for how long are new private keys valid before we replace them with a new one (which would in turn also live for TTL seconds before getting in turn replaced by a new one and so on)?

yup, that was my take: spec'ing it in relative time (rather than fixed epoch~ish),
that way it'd naturally go rotating[...] w/o further instrumentation.

@mkmik
Copy link
Collaborator Author

mkmik commented Jul 26, 2019

But that's already handled by the rotate-period setting. This issue is specifically about one-off operations

mkmik pushed a commit that referenced this issue Aug 5, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
mkmik pushed a commit that referenced this issue Aug 5, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
mkmik pushed a commit that referenced this issue Aug 5, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
mkmik pushed a commit that referenced this issue Aug 6, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
mkmik pushed a commit that referenced this issue Aug 6, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
mkmik pushed a commit that referenced this issue Aug 6, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
mkmik pushed a commit that referenced this issue Aug 6, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
mkmik pushed a commit that referenced this issue Aug 6, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
mkmik pushed a commit that referenced this issue Aug 7, 2019
Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
bors bot added a commit that referenced this issue Aug 7, 2019
214: Refactor key book-keeping r=mkmik a=mkmik

Prepares the ground for addressing #185 and #199 without adding any feature.

The only small side effect is that now private keys are attempted in random order (Go map iteration order).
This could only affect users who have turned on the key rotation feature which, as mentioned in #137, is not yet ready for prime time in the first place.
Also improves on #190.

Co-authored-by: Marko Mikulicic <mkm@bitnami.com>
This was referenced Oct 28, 2019
bors bot added a commit that referenced this issue Oct 29, 2019
299: Implement cutoff time r=mkmik a=mkmik

Addresses #199

TODO:
* some tests 
* update doc



Co-authored-by: Marko Mikulicic <mkm@bitnami.com>
bors bot added a commit that referenced this issue Oct 29, 2019
299: Implement cutoff time r=mkmik a=mkmik

Addresses #199

TODO:
* some tests 
* update doc



Co-authored-by: Marko Mikulicic <mkm@bitnami.com>
@github-actions
Copy link
Contributor

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

@github-actions github-actions bot added the Stale label Jan 28, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2022

Due to the lack of activity in the last 7 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

@github-actions github-actions bot closed this as completed Feb 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants