Description
Edit: this bug originally covered only the status of a waypoint not being updated, but on further investigation, it was discovered that this impacts all gateways, and causes them to be improperly programmed, not just to lack status.
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bugThis is not a question about how to use Istio
Bug Description
When using a gateway with an istio.io/rev label pointing to a tag rather than a revision, the gateway is ignored during the construction of the pushContext, causing incorrect XDS to be delivered to it, as well as preventing the status field from being set (if present). This issue affects both Istio Gateways and K8s Gateways across all methods of orchestration (helm, injection, managed, unmanaged).
Version
This issue was discovered on 1.24.1, but appears to have been present since tags were added to the codebase, sometime before 1.16.
client version: 1.24.1
control plane version: 1.24.1
data plane version: 1.24.1 (7 proxies)
Additional Information
to reproduce:
istioctl install --set profile=ambient --revision 1-24-1
istioctl tag set rapid --revision 1-24-1
istioctl waypoint apply --revision rapid --for all
Notice that status is not updated on the waypoint gateway resource. Additionally, run istioctl pc -o json against the gateway pod, and store this as tagged.json. Finally, run kubectl label deployment waypoint istio.io/rev=1-24-1 --overwrite
to use the explicit revision rather than the tag. Status should now be set, and the above pc command should have several thousand lines of significant difference frmo tagged.json.
Activity
therealmitchconnors commentedon Dec 23, 2024
This appears to be because the gateway status controller is listing gateways from the gateway controller, which is limited to gateways in this revision, while it is intended to write status for all gateways... that being said, I can't see how this ever worked, and I strongly suspect it worked in the past.
pilot/pkg/config/kube/crdclient/client.go:L71
keithmattix commentedon Dec 23, 2024
Even gateways that don't belong to this istiod's revision? That doesn't seem expected to me?
therealmitchconnors commentedon Jan 17, 2025
No, you are correct. Only gateways that belong to this revision of Istiod, based on their use of revision or tags in labels on the gateway or namespace.
therealmitchconnors commentedon Jan 17, 2025
In response to the question of how we could have not noticed this for so long, it appears most documentation about tag usage has historically pointed to using tags on namespaces, which still works. This bug is only triggered when a gateway object is explicitly labeled with a tag.
JanaSabuj commentedon Feb 13, 2025
I am facing a similar issue I think. I have the istio.io/rev=1-23 label on the gtw object and the waypoint that spawns up is not configured properly. I see that clusters and endpoints are missing (for the service I intended the waypoint to be for) in
istioctl pc clusters/endpoints <waypoint>
.Slack thread for the issue: https://istio.slack.com/archives/C041EQL1XMY/p1738912239987509