KedaProvider GetExternalMetric can return nil,nil and cause nil pointer reference #6009
Closed
Description
Report
Here in the the KedaProvider https://github.com/openshift/kedacore-keda/blob/be1b1737c53f708a62a6a7386b27f5f208bf7094/pkg/provider/provider.go#L92. The GetExternalMetric function can return nil,nil in the wait for the connection to be ready:
if !p.grpcClient.WaitForConnectionReady(ctx, logger) {
grpcClientConnected = false
logger.Error(fmt.Errorf("timeout while waiting to establish gRPC connection to KEDA Metrics Service server"), "timeout", "server", p.grpcClient.GetServerURL())
return nil, err
}
since it does not reassign the err variable.
This is reflected by a stack trace thrown in the keda metrics api-server pod below.
The nil,nil gets dereferenced here: https://github.com/kubernetes-sigs/custom-metrics-apiserver/blob/1e7ed5df24ef39898861260b002d9400f2d09300/pkg/registry/external_metrics/reststorage.go#L87 and hence panics in the code.
Expected Behavior
Expect the code to not panic.
Actual Behavior
The keda-api-metrics-server pod crashes.
Steps to Reproduce the Problem
Logs from KEDA operator
E0729 21:44:16.362477 1 timeout.go:142] post-timeout activity - time-elapsed: 85.660546ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/bpd-corebanking-pmt-transact-app/s0-prometheus" result: runtime error: invalid memory address or nil pointer dereference
goroutine 21782 [running]:
[k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1()](http://k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1())
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:110 +0x9c
panic({0x2684640?, 0x484a8f0?})
/usr/lib/golang/src/runtime/panic.go:914 +0x21f
[sigs.k8s.io/custom-metrics-apiserver/pkg/registry/external_metrics.(*REST).List(0xc0009a2480](http://sigs.k8s.io/custom-metrics-apiserver/pkg/registry/external_metrics.(*REST).List(0xc0009a2480), {0x2fe3038, 0xc0014d7170}, 0x0?)
/remote-source/keda/app/vendor/sigs.k8s.io/custom-metrics-apiserver/pkg/registry/external_metrics/reststorage.go:92 +0x112
[k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1({0x2fd3f20](http://k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(%7B0x2fd3f20)?, 0xc00123ef40}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:278 +0xe9c
[sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.(*EMHandlers).registerResourceHandlers.restfulListResource.func1(0xc00123ef20](http://sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.(*EMHandlers).registerResourceHandlers.restfulListResource.func1(0xc00123ef20), 0xc0004dd3b0)
/remote-source/keda/app/vendor/sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:274 +0x6a
[sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.(*EMHandlers).registerResourceHandlers.InstrumentRouteFunc.func2(0xc00123ef20](http://sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.(*EMHandlers).registerResourceHandlers.InstrumentRouteFunc.func2(0xc00123ef20), 0xc0004dd3b0)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:571 +0x1ce
[github.com/emicklei/go-restful/v3.(*Container).dispatch(0xc001568ea0](http://github.com/emicklei/go-restful/v3.(*Container).dispatch(0xc001568ea0), {0x2fd3f20, 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/github.com/emicklei/go-restful/v3/container.go:299 +0x5f0
[github.com/emicklei/go-restful/v3.(*Container).Dispatch(...)](http://github.com/emicklei/go-restful/v3.(*Container).Dispatch(...))
/remote-source/keda/app/vendor/github.com/emicklei/go-restful/v3/container.go:204
[k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x2b20adf](http://k8s.io/apiserver/pkg/server.director.ServeHTTP(%7B%7B0x2b20adf)?, 0x29b30a0?}, 0xc001568ea0?, 0xc0006d4930?}, {0x2fd3f20, 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x56c
[k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func21({0x2fd3f20](http://k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func21(%7B0x2fd3f20)?, 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
net/http.HandlerFunc.ServeHTTP(0x2fe3038?, {0x2fd3f20?, 0xc00123e140?}, 0x4?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/endpoints/filters.withAuthorization.func1({0x2fd3f20](http://k8s.io/apiserver/pkg/endpoints/filters.withAuthorization.func1(%7B0x2fd3f20), 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:78 +0x639
net/http.HandlerFunc.ServeHTTP(0xc1a2211d106f5039?, {0x2fd3f20?, 0xc00123e140?}, 0xc0009090b8?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x2fd3f20](http://k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(%7B0x2fd3f20), 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x192
net/http.HandlerFunc.ServeHTTP(0x4864600?, {0x2fd3f20?, 0xc00123e140?}, 0x4?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x2fd3f20](http://k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(%7B0x2fd3f20), 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:196 +0x262
net/http.HandlerFunc.ServeHTTP(0x1909f6f?, {0x2fd3f20?, 0xc00123e140?}, 0x2fab540?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func23({0x2fd3f20](http://k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func23(%7B0x2fd3f20)?, 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
net/http.HandlerFunc.ServeHTTP(0x1?, {0x2fd3f20?, 0xc00123e140?}, 0x1?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithImpersonation.func4({0x2fd3f20](http://k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithImpersonation.func4(%7B0x2fd3f20), 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1c3
net/http.HandlerFunc.ServeHTTP(0x829de0?, {0x2fd3f20?, 0xc00123e140?}, 0x0?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x2fd3f20](http://k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(%7B0x2fd3f20), 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x192
net/http.HandlerFunc.ServeHTTP(0x1909f6f?, {0x2fd3f20?, 0xc00123e140?}, 0x2fab540?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func24({0x2fd3f20](http://k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func24(%7B0x2fd3f20)?, 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
net/http.HandlerFunc.ServeHTTP(0x829de0?, {0x2fd3f20?, 0xc00123e140?}, 0x1?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x2fd3f20](http://k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(%7B0x2fd3f20), 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x192
net/http.HandlerFunc.ServeHTTP(0x1909f6f?, {0x2fd3f20?, 0xc00123e140?}, 0x2fab540?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func26({0x2fd3f20](http://k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func26(%7B0x2fd3f20)?, 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x177
net/http.HandlerFunc.ServeHTTP(0x2fe3038?, {0x2fd3f20?, 0xc00123e140?}, 0x2fa5230?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x2fd3f20](http://k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1(%7B0x2fd3f20), 0xc00123e140}, 0xc000e0cc00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:105 +0x6c5
net/http.HandlerFunc.ServeHTTP(0x2fe3038?, {0x2fd3f20?, 0xc00123e140?}, 0x2fab540?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x2fd3f20](http://k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(%7B0x2fd3f20), 0xc00123e140}, 0xc000e0ca00)
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:94 +0x37a
net/http.HandlerFunc.ServeHTTP(0xc000fb4f68?, {0x2fd3f20?, 0xc00123e140?}, 0x9e80cc?)
/usr/lib/golang/src/net/http/server.go:2136 +0x29
[k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()](http://k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1())
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:115 +0x62
created by [k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP](http://k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP) in goroutine 21781
/remote-source/keda/app/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101 +0x1b2
KEDA Version
2.12.1
Kubernetes Version
None
Platform
Red Hat OpenShift
Scaler Details
No response
Anything else?
The scaled object is of prometheus type:
triggers:
- metadata:
disableScaleToZero: 'false'
metricName: no_of_agents
query: no_of_agents
serverAddress: 'http://prometheussrv-exporter.bpd-corebanking-dev-transact-app.svc.cluster.local/'
threshold: '16'
type: prometheus