Document and (partially) handle interactions involving k8s terminationGracePeriodSeconds
#32487
Conversation
| Note that if running in Kubernetes, a SIGKILL will be issued after a set amount of time has passed | ||
| since SIGTERM. By default this time is 30 seconds ([`terminationGracePeriodSeconds`](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-execution)): | ||
| it may be worth adjusting the Kubernetes configuration or the phase timeouts to make `CoordinatedShutdown` | ||
| more likely to completely exectue before SIGKILL is received. |
There was a problem hiding this comment.
would it be worth changing the default-phase-timeout to 4 seconds so that our default totalTimeout is less than 30 seconds?
There was a problem hiding this comment.
Maybe not in akka-actor, which would apply to everyone wherever they're running. But it could be a responsibility of the Akka Management modules which imply k8s is in use set the default-phase-timeout like that.
There was a problem hiding this comment.
Crazy idea could be to have a mechanism to read the terminationGracePeriodSeconds from k8s and set the default timeout to floor(terminationGracePeriodSeconds / 7).seconds, but that would require setting the default timeout after starting the actorsystem.
There was a problem hiding this comment.
My read of reference.conf is that we have 11 levels of phases before before-actor-system-terminate (two of which have 10s phase timeouts). So only a default-phase-timeout of 1s would keep this below 30 seconds?
There was a problem hiding this comment.
On library (akka-management) can't override reference.conf of another library (akka-actor).
My point is that the value of default-phase-timeout was originally set to something arbitrary, and I can't imagine it would break any existing to change it to 4 seconds. We could do that separately for Akka 2.10.0 (and include note about config change in migration guide).
There was a problem hiding this comment.
Yeah, makes sense to do it for the next release.
By default Kubernetes will send SIGTERM, wait 30 seconds, and then send SIGKILL.
The SIGTERM will normally trigger CoordinatedShutdown, which triggers a shard rebalance. In the default configurations, the total CoordinatedShutdown timeout is greater than the k8s
terminationGracePeriodSecondsand less than the shardinghandoff-timeout.Adjusting the various phase timeouts to comport with the Kubernetes setting (or even just the default) seems dicey, so this just adds documentation to
CoordinatedShutdownsuggesting that one consider extendingterminationGracePeriodSeconds. It likewise documents that in the shutdown case, thehandoff-timeoutmay not apply.The behavior of the handoff now changes slightly to use a shorter effective handoff timeout when CoordinatedShutdown is in effect. The visible effect is that, when CoordinatedShutdown triggered the handoff, the warning about how long before entities will be forcibly stopped will be logged and the forcible stop may in fact happen at cluster sharding's hand rather than from the actor system itself.