Don't apply waitBeforeExitSeconds to control-plane pods
#10276
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Close #10058 The Helm value `proxy.waitBeforeExitSeconds` introduces a pause after the pod receives the shutdown signal, and it was intended for pods in the data-plane whose main container needs to perform shutdown-time operations that require the network. Linkerd's control-plane pods don't require that*. Additionally, if such shutdown operations take longer than 30s, then the user needs to set the pod's `terminationGracePeriod` (whose default is 30s) to be greater than `proxy.waitBeforeExitSeconds` to avoid the kubelet killing the pod before the operations completes. We don't expose `terminationGracePeriod` as a parameter to linkerd's pods so this scenario results in an error such as this: ``` Exec lifecycle hook ([/bin/sleep 40]) for Container "linkerd-proxy" in Pod "linkerd-destination-9559586c5-g9jns_linkerd(e33e8d02-66ca-42fa-9a7c-0ea45bda814a)" failed - error: command '/bin/sleep 40' exited with 137: , message: "" ``` For these two reasons, this change disables the `proxy.waitBeforeExitSeconds` setting for the linkerd pods, either by overriding it at the template level (for core control-plane pods) or through an annotation (for extension pods). (*) The Viz and Jaeger extensions don't require the network during shutdown either. The Multicluster extension already exposes a setting for `terminationGracePeriod`, so this change doesn't affect this particular extension.
alpeb
force-pushed
the
alpeb/terminationgraceperiod
branch
from
14c98de to
d3b4fd9
Compare
adleong
approved these changes
Feb 7, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Close #10058
The Helm value
proxy.waitBeforeExitSecondsintroduces a pause after the pod receives the shutdown signal, and it was intended for pods in the data-plane whose main container needs to perform shutdown-time operations that require the network. Linkerd's control-plane pods don't require that*.Additionally, if such shutdown operations take longer than 30s, then the user needs to set the pod's
terminationGracePeriod(whose default is 30s) to be greater thanproxy.waitBeforeExitSecondsto avoid the kubelet killing the pod before the operations completes. We don't exposeterminationGracePeriodas a parameter to linkerd's pods so this scenario results in an error such as this:For these two reasons, this change disables the
proxy.waitBeforeExitSecondssetting for the linkerd pods, either by overriding it at the template level (for core control-plane pods) or through an annotation (for extension pods).(*) The Viz and Jaeger extensions don't require the network during shutdown either. The Multicluster extension already exposes a setting for
terminationGracePeriod, so this change doesn't affect this particular extension.