Open
Description
Which tests failed?
Logs: https://github.com/solo-io/gloo/actions/runs/11533043315/job/32105508991?pr=10238
During the proxy syncer execution, we see:
{"level":"info","ts":"2024-10-26T15:55:14.423Z","logger":"k8s-gw-syncer","caller":"proxy_syncer/proxy_syncer.go:358","msg":"in gloo endpoints transformation"}
{"level":"error","ts":"2024-10-26T15:55:14.424Z","logger":"k8s-gw-syncer","caller":"proxy_syncer/proxy_syncer.go:551","msg":"upstream gloo-system.gateway-conformance-infra-grpc-infra-backend-v3-8080: port 8080 not found for service grpc-infra-backend-v3","stacktrace":"github.com/solo-io/gloo/projects/gateway2/proxy_syncer.buildEndpoints\n\t/home/runner/work/gloo/gloo/projects/gateway2/proxy_syncer/proxy_syncer.go:551\ngithub.com/solo-io/gloo/projects/gateway2/proxy_syncer.(*ProxySyncer).Start.func3\n\t/home/runner/work/gloo/gloo/projects/gateway2/proxy_syncer/proxy_syncer.go:361\nistio.io/istio/pkg/kube/krt.NewManyFromNothing[...].func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/singleton.go:203\nistio.io/istio/pkg/kube/krt.(*manyCollection[...]).onPrimaryInputEventLocked\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:252\nistio.io/istio/pkg/kube/krt.(*manyCollection[...]).onSecondaryDependencyEvent\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:512\nistio.io/istio/pkg/kube/krt.(*collectionDependencyTracker[...]).registerDependency.func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:633\nistio.io/istio/pkg/kube/krt.Fetch[...].func1.1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/fetch.go:47\nistio.io/istio/pkg/kube/krt.(*informer[...]).RegisterBatch.func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/informer.go:108\nistio.io/istio/pkg/kube/krt.(*informer[...].func4\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/informer.go:139\nistio.io/istio/pkg/kube/controllers.EventHandler[...].OnDelete\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/controllers/common.go:327\nk8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/controller.go:333\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/shared_informer.go:983\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:161\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/shared_informer.go:972\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/wait.go:72"}
{"level":"error","ts":"2024-10-26T15:55:14.424Z","logger":"k8s-gw-syncer","caller":"proxy_syncer/proxy_syncer.go:551","msg":"upstream gloo-system.gateway-conformance-infra-grpc-infra-backend-v2-8080: port 8080 not found for service grpc-infra-backend-v2","stacktrace":"github.com/solo-io/gloo/projects/gateway2/proxy_syncer.buildEndpoints\n\t/home/runner/work/gloo/gloo/projects/gateway2/proxy_syncer/proxy_syncer.go:551\ngithub.com/solo-io/gloo/projects/gateway2/proxy_syncer.(*ProxySyncer).Start.func3\n\t/home/runner/work/gloo/gloo/projects/gateway2/proxy_syncer/proxy_syncer.go:361\nistio.io/istio/pkg/kube/krt.NewManyFromNothing[...].func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/singleton.go:203\nistio.io/istio/pkg/kube/krt.(*manyCollection[...]).onPrimaryInputEventLocked\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:252\nistio.io/istio/pkg/kube/krt.(*manyCollection[...]).onSecondaryDependencyEvent\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:512\nistio.io/istio/pkg/kube/krt.(*collectionDependencyTracker[...]).registerDependency.func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:633\nistio.io/istio/pkg/kube/krt.Fetch[...].func1.1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/fetch.go:47\nistio.io/istio/pkg/kube/krt.(*informer[...]).RegisterBatch.func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/informer.go:108\nistio.io/istio/pkg/kube/krt.(*informer[...].func4\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/informer.go:139\nistio.io/istio/pkg/kube/controllers.EventHandler[...].OnDelete\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/controllers/common.go:327\nk8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/controller.go:333\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/shared_informer.go:983\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:161\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/shared_informer.go:972\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/wait.go:72"}
{"level":"error","ts":"2024-10-26T15:55:14.424Z","logger":"k8s-gw-syncer","caller":"proxy_syncer/proxy_syncer.go:551","msg":"upstream gateway-conformance-infra.kube-svc:gateway-conformance-infra-grpc-infra-backend-v2-8080: port 8080 not found for service grpc-infra-backend-v2","stacktrace":"github.com/solo-io/gloo/projects/gateway2/proxy_syncer.buildEndpoints\n\t/home/runner/work/gloo/gloo/projects/gateway2/proxy_syncer/proxy_syncer.go:551\ngithub.com/solo-io/gloo/projects/gateway2/proxy_syncer.(*ProxySyncer).Start.func3\n\t/home/runner/work/gloo/gloo/projects/gateway2/proxy_syncer/proxy_syncer.go:361\nistio.io/istio/pkg/kube/krt.NewManyFromNothing[...].func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/singleton.go:203\nistio.io/istio/pkg/kube/krt.(*manyCollection[...]).onPrimaryInputEventLocked\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:252\nistio.io/istio/pkg/kube/krt.(*manyCollection[...]).onSecondaryDependencyEvent\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:512\nistio.io/istio/pkg/kube/krt.(*collectionDependencyTracker[...]).registerDependency.func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/collection.go:633\nistio.io/istio/pkg/kube/krt.Fetch[...].func1.1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/fetch.go:47\nistio.io/istio/pkg/kube/krt.(*informer[...]).RegisterBatch.func1\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/informer.go:108\nistio.io/istio/pkg/kube/krt.(*informer[...].func4\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/krt/informer.go:139\nistio.io/istio/pkg/kube/controllers.EventHandler[...].OnDelete\n\t/home/runner/go/pkg/mod/istio.io/istio@v0.0.0-20240903161425-b8197f455bcf/pkg/kube/controllers/common.go:327\nk8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/controller.go:333\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/shared_informer.go:983\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:161\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/shared_informer.go:972\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/wait.go:72"}
Initial Investigation
This error occurs in a recently modified codepath, from when we introduced KRT (solo-io#10193). My first guess is that this error/flake is somehow related to those changes.
Additional Information
@lgadban How would you recommend users/devs debug a situation like this? I hope to look into this more to better understand KRT (and would happily pair on debugging this since you know more)
Would it be worth it to maintain the old strategy in code, and let users opt back into it? It would be an transitional state, and would enable us to also write tests comparing the outputs of both strategies as a way of identifying bugs.
Activity