KEDA tries to resolve secrets from restricted namespace #4519
Closed
Description
Report
I have KEDA up and running using KEDA_RESTRICT_SECRET_ACCESS environment variable which restricts secret access to only KEDA namespace. I have a rather simple ScaleObject (below) which has a target deployment and scale trigger using prometheus as a source. There is no TriggerAuthentication or any other reference to secrets in ScaledObject.
The target deployment (test-deployment) has some environment variables from secrets:
envFrom:
- secretRef:
name: test-deployment-89c6h5f565apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: test-scaledobject
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-deployment
pollingInterval: 30
minReplicaCount: 2
maxReplicaCount: 5
advanced:
restoreToOriginalReplicaCount: false
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 900
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.prometheus:9090/
query: max by(test_tasks_queue)(test_tasks_pending_blocked{service="test-deployment", test_tasks_queue="default"})
threshold: '100'
activationThreshold: '0'When I apply the ScaledObject to the cluster it fails to create the HPA resource for ScaledObject because it cannot find the secret what the deployment uses:
keda-operator error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:test-deployment-89c6h5f565,},Optional:nil,} on namespace default: secret "test-deployment-89c6h5f565" not found
keda-operator Failed to ensure HPA is correctly created for ScaledObject
However if I allow KEDA to access secrets from all namespaces the ScaledObject and HPA are created succesfully and everything works fine.
Expected Behavior
KEDA should not need to access secrets from the namespace where the scale target is located if there is no reference to a secret in the ScaledObject or TriggerAuthentication is not used.
Actual Behavior
KEDA tries to list/watch/get secrets from the scale target namespace even though usage of secret(s) is not required from ScaledObject.
Steps to Reproduce the Problem
- KEDA is installed using helm and the access to secrets is limited using the following parameters in values.yaml
permissions:
metricServer:
restrict:
secret: true
operator:
restrict:
secret: true- ScaledObject (described in the report section) is applied to the cluster
- Creation of HPA resource for ScaledObject fails
Logs from KEDA operator
"level":"error","ts":"2023-05-03T10:57:16Z","msg":"Failed to ensure HPA is correctly created for ScaledObject","controller":"scaledobject","controllerGroup":"keda.sh","controllerKind":"ScaledObject","ScaledObject":{"name":"test-scaledobject","namespace":"default"},"namespace":"default","name":"test-scaledobject","reconcil │
eID":"7fc01087-cd26-479e-b31c-567dba2736e1","error":"error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:test-deployment-89c6h5f565,},Optional:nil,} on namespace default: secret \"test-deployment-89c6h5f565\" not found","stacktrace":"github.com/kedacore/keda/v2/contro │
llers/keda.(*ScaledObjectReconciler).Reconcile\n\t/workspace/controllers/keda/scaledobject_controller.go:178\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconci │
leHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).St │
art.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-05-03T10:57:16Z","msg":"Reconciler error","controller":"scaledobject","controllerGroup":"keda.sh","controllerKind":"ScaledObject","ScaledObject":{"name":"test-scaledobject","namespace":"default"},"namespace":"default","name":"test-scaledobject","reconcileID":"7fc01087-cd26-479e-b31c-567dba2736e1 │
","error":"error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:test-deployment-89c6h5f565,},Optional:nil,} on namespace default: secret \"test-deployment-89c6h5f565\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconci │
leHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).St │
art.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}
KEDA Version
2.10.0
Kubernetes Version
1.24
Platform
Google Cloud
Scaler Details
Prometheus
Anything else?
No response
Metadata
Assignees
Type
Projects
Status
Ready To Ship