istio-operator/istioctl > 1.19 unexpectedly remove MutatingWebhookConfiguration::istio-revision-tag #47423
Description
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
We are following Istio Canary Upgrades and found istioctl uninstall will unexpected remove the revision-tag created for canary upgrades.
Version
$ istioctl version
client version: 1.18.1
control plane version: 1.18.1
data plane version: 1.18.1 (1 proxies)
$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.6
WARNING: version difference between client (1.28) and server (1.26) exceeds the supported minor version skew of +/-1
Additional Information
How to repro
Install Istio 1-18-1
$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.18.1 sh -
$ istioctl version
no ready Istio pods in "istio-system"
1.18.1
$ istioctl operator init --revision 1-18-1 --tag "1.18.1-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.18.1-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete
$ cat istio-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
profile: minimal
tag: 1.18.1-distroless
revision: 1-18-1
components:
pilot:
enabled: true
ingressGateways:
- enabled: true
name: istio-ingressgateway
$ k apply -f istio-operator.yaml
istiooperator.install.istio.io/istiocontrolplane created
$ k label namespace default istio.io/rev=prod-stable --overwrite
namespace/default labeled
$ istioctl tag set prod-canary --revision 1-18-1
Revision tag "prod-canary" created, referencing control plane revision "1-18-1". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'
$ istioctl tag set prod-stable --revision 1-18-1
Revision tag "prod-stable" created, referencing control plane revision "1-18-1". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'
$ istioctl tag list
TAG REVISION NAMESPACES
prod-canary 1-18-1
prod-stable 1-18-1 default
$ k get MutatingWebhookConfigurations
NAME WEBHOOKS AGE
istio-revision-tag-prod-canary 2 27s
istio-revision-tag-prod-stable 2 18s
istio-sidecar-injector-1-18-1 2 13m
Upgrade to istio 1-19-3
$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.19.3 sh -
$ istioctl version
client version: 1.19.3
control plane version: 1.18.1
data plane version: 1.18.1 (1 proxies)
$ istioctl operator init --revision 1-19-3 --tag "1.19.3-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.19.3-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete
$ cat istio-operator-1.19.3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
profile: minimal
tag: 1.19.3-distroless
revision: 1-19-3
components:
pilot:
enabled: true
ingressGateways:
- enabled: true
name: istio-ingressgateway
$ k apply -f istio-operator-1.19.3.yaml
istiooperator.install.istio.io/istiocontrolplane configured
$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'
$ istioctl tag set prod-canary --revision 1-19-3 --overwrite
Revision tag "prod-canary" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'
$ istioctl tag list
TAG REVISION NAMESPACES
prod-canary 1-19-3
prod-stable 1-19-3 default
Uninstall Istio 1-18-1
$ istioctl uninstall --revision 1-18-1
Removed Deployment:istio-operator:istio-operator-1-18-1.
Removed Deployment:istio-system:istiod-1-18-1.
Removed Service:istio-operator:istio-operator-1-18-1.
Removed Service:istio-system:istiod-1-18-1.
Removed ConfigMap:istio-system:istio-1-18-1.
Removed ConfigMap:istio-system:istio-sidecar-injector-1-18-1.
object: Pod:istio-system:istiod-1-18-1-d4b874889-mbdqx is not being deleted because it no longer exists
Removed ServiceAccount:istio-operator:istio-operator-1-18-1.
Removed ServiceAccount:istio-system:istiod-1-18-1.
Removed RoleBinding:istio-system:istiod-1-18-1.
Removed Role:istio-system:istiod-1-18-1.
Removed HorizontalPodAutoscaler:istio-system:istiod-1-18-1.
Removed PodDisruptionBudget:istio-system:istiod-1-18-1.
Removed MutatingWebhookConfiguration::istio-sidecar-injector-1-18-1.
Removed ValidatingWebhookConfiguration::istio-validator-1-18-1-istio-system.
Removed ClusterRole::istio-operator-1-18-1.
Removed ClusterRole::istio-reader-clusterrole-1-18-1-istio-system.
Removed ClusterRole::istiod-clusterrole-1-18-1-istio-system.
Removed ClusterRole::istiod-gateway-controller-1-18-1-istio-system.
Removed ClusterRoleBinding::istio-operator-1-18-1.
Removed ClusterRoleBinding::istio-reader-clusterrole-1-18-1-istio-system.
Removed ClusterRoleBinding::istiod-clusterrole-1-18-1-istio-system.
Removed ClusterRoleBinding::istiod-gateway-controller-1-18-1-istio-system.
After above command run, istio-operator log shows istio-operator deleted the revision-tag
Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
✔ Ingress gateways installed
- Pruning removed resources[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
> goroutine 411 [running]:
> runtime/debug.Stack()
> runtime/debug/stack.go:24 +0x5e
> sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
> sigs.k8s.io/controller-runtime@v0.16.0/pkg/log/log.go:60 +0xcd
> sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc001531780, 0x0?)
> sigs.k8s.io/controller-runtime@v0.16.0/pkg/log/deleg.go:111 +0x32
> github.com/go-logr/logr.Logger.Enabled(...)
> github.com/go-logr/logr@v1.2.4/logr.go:261
> github.com/go-logr/logr.Logger.Info({{0x3449f28?, 0xc001531780?}, 0xc000eb5ad0?}, {0x2eaaa40, 0x14}, {0x0, 0x0, 0x0})
> github.com/go-logr/logr@v1.2.4/logr.go:274 +0x72
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000c16be0, {0x3443ec8, 0xc000ae5ef0}, {0x2a90f80?, 0xc0000b7bc0?})
> sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:344 +0x4f7
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000c16be0, {0x3443ec8, 0xc000ae5ef0})
> sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:266 +0x1c9
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
> sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:227 +0x79
> created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 73
> sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:223 +0x565
2023-10-18T05:59:43.118237Z info installer Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:43.118344Z info installer Reconciling IstioOperator
2023-10-18T05:59:43.172465Z info installer Updating IstioOperator
2023-10-18T05:59:43.172841Z info installer Detecting third-party JWT support
2023-10-18T05:59:43.663698Z info installer Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:43.964206Z info installer Watching a change for istio resource: istio-system/istio-1-18-1
2023-10-18T05:59:44.276378Z info installer Watching a change for istio resource: istio-system/istio-sidecar-injector-1-18-1
2023-10-18T05:59:44.784728Z info kube Starting Pilot K8S CRD controller controller=analysis-controller
2023-10-18T05:59:44.784764Z info kube Pilot K8S CRD controller synced in 43.103µs controller=analysis-controller
2023-10-18T05:59:44.784780Z info kube Starting Pilot K8S CRD controller controller=analysis-controller
2023-10-18T05:59:44.784789Z info kube Pilot K8S CRD controller synced in 11.401µs controller=analysis-controller
2023-10-18T05:59:44.787734Z info installer Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T05:59:44.787781Z info installer Pilot is waiting on dependency...
2023-10-18T05:59:44.787797Z info installer Cni is waiting on dependency...
2023-10-18T05:59:44.787807Z info installer IngressGateways is waiting on dependency...
2023-10-18T05:59:44.787815Z info installer Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T05:59:44.787868Z info installer Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T05:59:44.787879Z info installer EgressGateways is waiting on dependency...
2023-10-18T05:59:44.787744Z info installer Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T05:59:44.788452Z info installer Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T05:59:45.064198Z info installer Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.368543Z info installer Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.524126Z info installer Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.864460Z info installer Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:46.055691Z info installer Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:46.074125Z info installer Generated manifest objects are the same as cached for component Base.
2023-10-18T05:59:46.074185Z info installer Unblocking dependency Pilot.
2023-10-18T05:59:46.074224Z info installer Dependency for Pilot has completed, proceeding.
2023-10-18T05:59:46.074234Z info installer Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
2023-10-18T05:59:46.269846Z info installer Generated manifest objects are the same as cached for component Pilot.
2023-10-18T05:59:46.270075Z info installer Unblocking dependency Cni.
2023-10-18T05:59:46.270149Z info installer Unblocking dependency IngressGateways.
2023-10-18T05:59:46.270190Z info installer Unblocking dependency EgressGateways.
2023-10-18T05:59:46.363699Z info installer Dependency for EgressGateways has completed, proceeding.
2023-10-18T05:59:46.363748Z info installer Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T05:59:46.363867Z info installer Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T05:59:46.363883Z info installer Dependency for Cni has completed, proceeding.
2023-10-18T05:59:46.363900Z info installer Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T05:59:46.363942Z info installer Generated manifest objects are the same as cached for component Cni.
2023-10-18T05:59:46.363948Z info installer Dependency for IngressGateways has completed, proceeding.
2023-10-18T05:59:46.363957Z info installer Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T05:59:46.367348Z info installer Watching a change for istio resource: /istio-sidecar-injector-1-18-1
2023-10-18T05:59:46.375135Z info installer Generated manifest objects are the same as cached for component IngressGateways.
2023-10-18T05:59:46.764964Z info installer Watching a change for istio resource: /istio-validator-1-18-1-istio-system
2023-10-18T05:59:47.166836Z info installer Watching a change for istio resource: /istio-reader-clusterrole-1-18-1-istio-system
2023-10-18T05:59:47.383171Z info installer Watching a change for istio resource: /istiod-clusterrole-1-18-1-istio-system
2023-10-18T05:59:47.672115Z info installer Watching a change for istio resource: /istiod-gateway-controller-1-18-1-istio-system
2023-10-18T05:59:48.178424Z info installer Watching a change for istio resource: /istio-reader-clusterrole-1-18-1-istio-system
2023-10-18T05:59:48.568199Z info installer Watching a change for istio resource: /istiod-clusterrole-1-18-1-istio-system
2023-10-18T05:59:48.709472Z info installer Watching a change for istio resource: /istiod-gateway-controller-1-18-1-istio-system
2023-10-18T06:00:01.971533Z info installer Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-canary from Cache.
Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:00:01.971583Z info Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:00:01.973419Z info installer Watching a change for istio resource: /istio-revision-tag-prod-canary
2023-10-18T06:00:01.978272Z info installer Watching a change for istio resource: /istio-revision-tag-prod-stable
2023-10-18T06:00:01.978330Z info installer Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-stable from Cache.
Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
2023-10-18T06:00:01.978365Z info Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
2023-10-18T06:00:04.586509Z info installer Reconciling IstioOperator
2023-10-18T06:00:04.680744Z info installer Updating IstioOperator
2023-10-18T06:00:04.680811Z info installer Detecting third-party JWT support
2023-10-18T06:00:06.371940Z info kube Starting Pilot K8S CRD controller controller=analysis-controller
2023-10-18T06:00:06.371984Z info kube Pilot K8S CRD controller synced in 51.203µs controller=analysis-controller
2023-10-18T06:00:06.372005Z info kube Starting Pilot K8S CRD controller controller=analysis-controller
2023-10-18T06:00:06.372015Z info kube Pilot K8S CRD controller synced in 14.2µs controller=analysis-controller
2023-10-18T06:00:06.374930Z info installer Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T06:00:06.375007Z info installer Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T06:00:06.375030Z info installer IngressGateways is waiting on dependency...
2023-10-18T06:00:06.375042Z info installer EgressGateways is waiting on dependency...
2023-10-18T06:00:06.375050Z info installer Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T06:00:06.467573Z info installer Cni is waiting on dependency...
2023-10-18T06:00:06.467621Z info installer Pilot is waiting on dependency...
2023-10-18T06:00:06.467644Z info installer Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T06:00:06.467706Z info installer Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T06:00:07.473118Z info installer Generated manifest objects are the same as cached for component Base.
2023-10-18T06:00:07.473176Z info installer Unblocking dependency Pilot.
2023-10-18T06:00:07.473196Z info installer Dependency for Pilot has completed, proceeding.
2023-10-18T06:00:07.473206Z info installer Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
2023-10-18T06:00:07.765838Z info installer Generated manifest objects are the same as cached for component Pilot.
2023-10-18T06:00:07.765881Z info installer Unblocking dependency Cni.
2023-10-18T06:00:07.765887Z info installer Unblocking dependency IngressGateways.
2023-10-18T06:00:07.765891Z info installer Unblocking dependency EgressGateways.
2023-10-18T06:00:07.765898Z info installer Dependency for EgressGateways has completed, proceeding.
2023-10-18T06:00:07.765908Z info installer Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T06:00:07.765953Z info installer Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T06:00:07.765959Z info installer Dependency for Cni has completed, proceeding.
2023-10-18T06:00:07.765964Z info installer Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T06:00:07.765989Z info installer Generated manifest objects are the same as cached for component Cni.
2023-10-18T06:00:07.765994Z info installer Dependency for IngressGateways has completed, proceeding.
2023-10-18T06:00:07.765998Z info installer Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T06:00:07.772990Z info installer Generated manifest objects are the same as cached for component IngressGateways.
- Pruning removed resources
$ istioctl tag list
No Istio revision tag MutatingWebhookConfigurations to list