PILOT_SEND_UNHEALTHY_ENDPOINTS: exclude terminating endpoints #54711
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The purpose of PILOT_SEND_UNHEALTHY_ENDPOINTS is to let envoy know about the count of health and unhealth endpoints as a factor into its failover decisions. For instance, if we have 5/10 pods health, it would failover. This has problematic behavior during scaling up/down. For instance, if I just have a single pod and then make a change, I will temporarily have 2 pods (the new one, and the old one terminating). Envoy will now see this as 50% health and failover. This is bad -- its not actually unhealthy at all! Additionally, when pods are scaling down we end up keeping them as unhealthy. Consider I have 1/2 pods healthy, and shut down the unhealthy one but it takes a few minutes to fully terminate. During that time, envoy will consider us 50% healthy. However, because we requested 1 pod and have 1 pod, I would consider that 100% healthy. ---- This PR adds a new health mode, "terminating". * When the feature flag is false, terminating and unhealthy mean the same thing -- do not send the endpoint to envoy. This has no impact. * When the feature flag is true, we will not sending terminating endpoints. This means a terminating endpoint is not taken into account for the health calculations. This change does not impact whether we send traffic to the terminating pods or not -- both before and after, we never do this. Note: there is another status, 'draining', which is a very specialized mode that is only used when session persistence is enabled on a service && the endpoint is terminating && it used to be healthy. Comments in the code clarify the differences between these two similar states. These endpoints *are* sent to envoy and traffic is sent to them, so its fundamentally different than Terminating.
howardjohn
added
the
release-notes-none
istio-testing
added
the
size/M
keithmattix
reviewed
Jan 16, 2025
bleggett
reviewed
Jan 16, 2025
I think the main cases are New rollout: Scaling: |
howardjohn
force-pushed
the
endpoints/terminating-mode
branch
from
4cc7579 to
4c55534
Compare
bleggett
approved these changes
Jan 16, 2025
keithmattix
approved these changes
Jan 16, 2025
keithmattix
added
do-not-merge/hold
|
This LGTM, but I agree with @bleggett on the release note. Adding a hold until that's done |
fjglira
pushed a commit
to fjglira/istio
that referenced
this pull request
Feb 10, 2025
* upstream/master: make collections implement Syncer (istio#54750) Automator: update proxy@master in istio/istio@master (istio#54755) istioctl: support SA RBAC rule parsing (istio#54744) Automator: update proxy@master in istio/istio@master (istio#54754) Automator: update proxy@master in istio/istio@master (istio#54753) Automator: update ztunnel@master in istio/istio@master (istio#54749) Enable modifying the prow cluster name (istio#54742) Automator: update proxy@master in istio/istio@master (istio#54727) Fix crash when starting pilot without kubeClient (istio#54730) ambient: support `istio.io/ingress-use-waypoint` at the namespace level (istio#54483) Automator: update ztunnel@master in istio/istio@master (istio#54733) run make gen (istio#54728) Be more explicit about how Helm value deprecations should manifest in relnotes (istio#54697) PILOT_SEND_UNHEALTHY_ENDPOINTS: exclude terminating endpoints (istio#54711) Fix ambient telemetry in ingress gateways (istio#54383) Automator: update proxy@master in istio/istio@master (istio#54713) Gateway also needs global flattening (istio#54714) Automator: update common-files@master in istio/istio@master (istio#54712) Automator: update proxy@master in istio/istio@master (istio#54709) Do not impersonate empty group (istio#54667)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The purpose of PILOT_SEND_UNHEALTHY_ENDPOINTS is to let envoy know about
the count of health and unhealth endpoints as a factor into its failover
decisions. For instance, if we have 5/10 pods health, it would failover.
This has problematic behavior during scaling up/down. For instance, if I
just have a single pod and then make a change, I will temporarily have 2
pods (the new one, and the old one terminating). Envoy will now see this
as 50% health and failover. This is bad -- its not actually unhealthy at
all!
Additionally, when pods are scaling down we end up keeping them as
unhealthy. Consider I have 1/2 pods healthy, and shut down the unhealthy
one but it takes a few minutes to fully terminate. During that time,
envoy will consider us 50% healthy. However, because we requested 1 pod
and have 1 pod, I would consider that 100% healthy.
This PR adds a new health mode, "terminating".
same thing -- do not send the endpoint to envoy. This has no impact.
endpoints. This means a terminating endpoint is not taken into account
for the health calculations.
This change does not impact whether we send traffic to the terminating
pods or not -- both before and after, we never do this.
Note: there is another status, 'draining', which is a very specialized
mode that is only used when session persistence is enabled on a service
&& the endpoint is terminating && it used to be healthy. Comments in the
code clarify the differences between these two similar states. These
endpoints are sent to envoy and traffic is sent to them, so its
fundamentally different than Terminating.