Skip to content

DataDog scaler makes errors in keda-metrics-apiserver and causes keda-operator crashing loop back #3448

New issue
Jump to bottom
New issue
Jump to bottom
Closed
#3450
Closed
DataDog scaler makes errors in keda-metrics-apiserver and causes keda-operator crashing loop back#3448
#3450
Assignees
JorTurFer
Labels
bugSomething isn't working
@dogzzdogzz

Description

@dogzzdogzz
dogzzdogzz
opened on Jul 28, 2022

Report

When I use the query sum:trace.express.request.hits{*}.as_rate()/avg:kubernetes.cpu.requests{*} in datadog scaler, the metrics-apiserver will have panic: runtime error: invalid memory address or nil pointer dereference and causes keda-operator crashing

Here is my scaledobject manifest

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: test
spec:
  cooldownPeriod: 10
  maxReplicaCount: 2
  minReplicaCount: 1
  pollingInterval: 30
  scaleTargetRef:
    kind: Deployment
    name: foo
  triggers:
  - authenticationRef:
      kind: ClusterTriggerAuthentication
      name: datadog-secret
    metadata:
      age: "120"
      metricUnavailableValue: "0"
      name: test3
      query: sum:trace.express.request.hits{*}.as_rate()/avg:kubernetes.cpu.requests{*}
      queryValue: "300"
    metricType: AverageValue
    type: datadog

I already confirmed that the query works without problem on datadog UI and with curl command

curl -X GET 'https://api.datadoghq.com/api/v1/query?from=1658854540&to=1658854660&query=sum:trace.express.request.hits\{*\}.as_rate()/avg:kubernetes.cpu.requests\{*\}' \
-H "Accept: application/json" \
-H "DD-API-KEY: XXXXX" \
-H "DD-APPLICATION-KEY: XXXXX"

Expected Behavior

The query sum:trace.express.request.hits{*}.as_rate()/avg:kubernetes.cpu.requests{*} returns the result as expected

Actual Behavior

metric-apiserver has errors and operator start crashing

Steps to Reproduce the Problem

If there is no as_rate() in query, it works without problem
sum:trace.express.request.hits{*}/avg:kubernetes.cpu.requests{*}

but as long as I add the as_rate() as below, the metrics-apiserver starts returning error and keda-operator start crashing
sum:trace.express.request.hits{*}.as_rate()/avg:kubernetes.cpu.requests{*}

Logs from KEDA operator

keda-metrics-apiserver

E0728 16:51:09.496563       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
goroutine 709876 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1()
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/timeout.go:109 +0xb0
panic({0x34aaa80, 0x65f1e90})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/kedacore/keda/v2/pkg/scalers.(*datadogScaler).getQueryResult(0xc00b31b8c0, {0x427bc70, 0xc00a16c030})
	/workspace/pkg/scalers/datadog_scaler.go:287 +0x696
github.com/kedacore/keda/v2/pkg/scalers.(*datadogScaler).GetMetrics(0xc00b31b8c0, {0x427bc70, 0xc00a16c030}, {0x16, 0xc008d86000}, {0x16, 0xc00b307840})
	/workspace/pkg/scalers/datadog_scaler.go:306 +0x46
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsForScaler(0xc008d402d0, {0x427bc70, 0xc00a16c030}, 0x0, {0xc008641f76, 0x29}, {0x42b5870, 0xc00b3ed800})
	/workspace/pkg/scaling/cache/scalers_cache.go:70 +0x136
github.com/kedacore/keda/v2/pkg/provider.(*KedaProvider).GetExternalMetric(0xc0000a5400, {0x427bc70, 0xc00a16c030}, {0xc008641f71, 0x4}, {0x42b5870, 0xc00b3ed800}, {{0xc008641f76, 0x3b2f613}})
	/workspace/pkg/provider/provider.go:119 +0x9d3
sigs.k8s.io/custom-metrics-apiserver/pkg/registry/external_metrics.(*REST).List(0xc000651a40, {0x427bc70, 0xc00a16c030}, 0xc009c8b290)
	/go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.23.0/pkg/registry/external_metrics/reststorage.go:80 +0x10a
k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1({0x4265d50, 0xc00b5780a0}, 0xc009c81700)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/handlers/get.go:278 +0x1136
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResource.func1(0xc00c407f50, 0xc0005f68c0)
	/go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.23.0/pkg/apiserver/installer/installer.go:291 +0x6b
k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc00c407f50, 0xc0005f68c0)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/metrics/metrics.go:509 +0x223
github.com/emicklei/go-restful.(*Container).dispatch(0xc000a073b0, {0x4265d50, 0xc00af78840}, 0xc009c81700)
	/go/pkg/mod/github.com/emicklei/go-restful@v2.15.0+incompatible/container.go:294 +0x690
github.com/emicklei/go-restful.(*Container).Dispatch(...)
	/go/pkg/mod/github.com/emicklei/go-restful@v2.15.0+incompatible/container.go:204
k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x3b23bef, 0x41dc870}, 0xc000a073b0, 0xc0001cee00}, {0x4265d50, 0xc00af78840}, 0xc009c81700)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/handler.go:146 +0x5c6
k8s.io/apiserver/pkg/endpoints/filters.WithWebhookDuration.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/webhook_duration.go:31 +0x24e
net/http.HandlerFunc.ServeHTTP(0xc00b14c420, {0x4265d50, 0xc00af78840}, 0xc1678a)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
net/http.HandlerFunc.ServeHTTP(0xc00111a300, {0x4265d50, 0xc00af78840}, 0xc009772340)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/authorization.go:64 +0x41d
net/http.HandlerFunc.ServeHTTP(0x66785b8, {0x4265d50, 0xc00af78840}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
net/http.HandlerFunc.ServeHTTP(0x66079a0, {0x4265d50, 0xc00af78840}, 0x4)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/maxinflight.go:187 +0x29d
net/http.HandlerFunc.ServeHTTP(0xc00b14c420, {0x4265d50, 0xc00af78840}, 0xc1678a)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
net/http.HandlerFunc.ServeHTTP(0x44e712, {0x4265d50, 0xc00af78840}, 0x1)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/impersonation.go:50 +0x21c
net/http.HandlerFunc.ServeHTTP(0x66785b8, {0x4265d50, 0xc00af78840}, 0xa)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
net/http.HandlerFunc.ServeHTTP(0xc00b14c420, {0x4265d50, 0xc00af78840}, 0xc1678a)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
net/http.HandlerFunc.ServeHTTP(0x66785b8, {0x4265d50, 0xc00af78840}, 0xa)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
net/http.HandlerFunc.ServeHTTP(0xc00b14c420, {0x4265d50, 0xc00af78840}, 0xc1678a)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
net/http.HandlerFunc.ServeHTTP(0x427bc70, {0x4265d50, 0xc00af78840}, 0x3c75060)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/authentication.go:80 +0x839
net/http.HandlerFunc.ServeHTTP(0x427bc38, {0x4265d50, 0xc00af78840}, 0x41dc870)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4265d50, 0xc00af78840}, 0xc00a064700)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:89 +0x46b
net/http.HandlerFunc.ServeHTTP(0xc009537380, {0x4265d50, 0xc00af78840}, 0x8e74f2)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/timeout.go:114 +0x70
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/timeout.go:100 +0x1dd

goroutine 709566 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x32b8940, 0xc00b4c9d60})
	/go/pkg/mod/k8s.io/apimachinery@v0.23.6/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc00121d088, 0x1, 0x17a0d66})
	/go/pkg/mod/k8s.io/apimachinery@v0.23.6/pkg/util/runtime/runtime.go:48 +0x75
panic({0x32b8940, 0xc00b4c9d60})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc00111a420, {0x4258e50, 0xc00064d990}, 0xdf8475800)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/timeout.go:120 +0x358
k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x4258e50, 0xc00064d990}, 0xc00a064500)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/request_deadline.go:100 +0x494
net/http.HandlerFunc.ServeHTTP(0x18, {0x4258e50, 0xc00064d990}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x4258e50, 0xc00064d990}, 0xc00a064500)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/waitgroup.go:77 +0x766
net/http.HandlerFunc.ServeHTTP(0x427bc70, {0x4258e50, 0xc00064d990}, 0x41dc870)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x4258e50, 0xc00064d990}, 0xc00a064400)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/warning.go:35 +0x2bb
net/http.HandlerFunc.ServeHTTP(0x373acc0, {0x4258e50, 0xc00064d990}, 0xd)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x4258e50, 0xc00064d990}, 0x3)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/cachecontrol.go:31 +0x126
net/http.HandlerFunc.ServeHTTP(0x203002, {0x4258e50, 0xc00064d990}, 0x30)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x4258e50, 0xc00064d990}, 0xc00a064400)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/httplog/httplog.go:108 +0xaf
net/http.HandlerFunc.ServeHTTP(0x427bc70, {0x4258e50, 0xc00064d990}, 0x41dc870)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x4258e50, 0xc00064d990}, 0xc00a064300)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/requestinfo.go:39 +0x316
net/http.HandlerFunc.ServeHTTP(0x427bc70, {0x4258e50, 0xc00064d990}, 0x2c)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x4258e50, 0xc00064d990}, 0xc00a064200)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/request_received_time.go:38 +0x27e
net/http.HandlerFunc.ServeHTTP(0x203002, {0x4258e50, 0xc00064d990}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x4258e50, 0xc00064d990}, 0xc00a064200)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x2c2
net/http.HandlerFunc.ServeHTTP(0x160, {0x4258e50, 0xc00064d990}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x4258e50, 0xc00064d990}, 0xc00a032f20)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/wrap.go:74 +0xba
net/http.HandlerFunc.ServeHTTP(0x373acc0, {0x4258e50, 0xc00064d990}, 0x8)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withAuditID.func1({0x4258e50, 0xc00064d990}, 0xc009c81600)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/with_auditid.go:66 +0x40d
net/http.HandlerFunc.ServeHTTP(0xc000e55d60, {0x4258e50, 0xc00064d990}, 0xc000e55ccc)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc008641fa0, {0x4258e50, 0xc00064d990}, 0x427bc00)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/handler.go:189 +0x2b
net/http.serverHandler.ServeHTTP({0xc000e55d18}, {0x4258e50, 0xc00064d990}, 0xc009c81600)
	/usr/local/go/src/net/http/server.go:2879 +0x43b
net/http.initALPNRequest.ServeHTTP({{0x427bc70, 0xc0096623f0}, 0xc000b3c000, {0xc0001b27e0}}, {0x4258e50, 0xc00064d990}, 0xc009c81600)
	/usr/local/go/src/net/http/server.go:3480 +0x245
golang.org/x/net/http2.(*serverConn).runHandler(0xc000e55fd0, 0x8f288a, 0x0, 0xc000e55eb8)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20220412020605-290c469a71a5/http2/server.go:2184 +0x78
created by golang.org/x/net/http2.(*serverConn).processHeaders
	/go/pkg/mod/golang.org/x/net@v0.0.0-20220412020605-290c469a71a5/http2/server.go:1914 +0x590
E0728 16:51:09.496619       1 wrap.go:58] "apiserver panic'd" method="GET" URI="/apis/external.metrics.k8s.io/v1beta1/namespaces/keda/s0-datadog-sum-trace-express-request-hits?labelSelector=scaledobject.keda.sh%2Fname%3Dtmp-shell" audit-ID="c4342740-4286-448c-bfb6-5e578dd8c5a4"
http2: panic serving 10.33.78.156:53522: runtime error: invalid memory address or nil pointer dereference
goroutine 709876 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1()
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/timeout.go:109 +0xb0
panic({0x34aaa80, 0x65f1e90})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/kedacore/keda/v2/pkg/scalers.(*datadogScaler).getQueryResult(0xc00b31b8c0, {0x427bc70, 0xc00a16c030})
	/workspace/pkg/scalers/datadog_scaler.go:287 +0x696
github.com/kedacore/keda/v2/pkg/scalers.(*datadogScaler).GetMetrics(0xc00b31b8c0, {0x427bc70, 0xc00a16c030}, {0x16, 0xc008d86000}, {0x16, 0xc00b307840})
	/workspace/pkg/scalers/datadog_scaler.go:306 +0x46
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsForScaler(0xc008d402d0, {0x427bc70, 0xc00a16c030}, 0x0, {0xc008641f76, 0x29}, {0x42b5870, 0xc00b3ed800})
	/workspace/pkg/scaling/cache/scalers_cache.go:70 +0x136
github.com/kedacore/keda/v2/pkg/provider.(*KedaProvider).GetExternalMetric(0xc0000a5400, {0x427bc70, 0xc00a16c030}, {0xc008641f71, 0x4}, {0x42b5870, 0xc00b3ed800}, {{0xc008641f76, 0x3b2f613}})
	/workspace/pkg/provider/provider.go:119 +0x9d3
sigs.k8s.io/custom-metrics-apiserver/pkg/registry/external_metrics.(*REST).List(0xc000651a40, {0x427bc70, 0xc00a16c030}, 0xc009c8b290)
	/go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.23.0/pkg/registry/external_metrics/reststorage.go:80 +0x10a
k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1({0x4265d50, 0xc00b5780a0}, 0xc009c81700)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/handlers/get.go:278 +0x1136
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResource.func1(0xc00c407f50, 0xc0005f68c0)
	/go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.23.0/pkg/apiserver/installer/installer.go:291 +0x6b
k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc00c407f50, 0xc0005f68c0)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/metrics/metrics.go:509 +0x223
github.com/emicklei/go-restful.(*Container).dispatch(0xc000a073b0, {0x4265d50, 0xc00af78840}, 0xc009c81700)
	/go/pkg/mod/github.com/emicklei/go-restful@v2.15.0+incompatible/container.go:294 +0x690
github.com/emicklei/go-restful.(*Container).Dispatch(...)
	/go/pkg/mod/github.com/emicklei/go-restful@v2.15.0+incompatible/container.go:204
k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x3b23bef, 0x41dc870}, 0xc000a073b0, 0xc0001cee00}, {0x4265d50, 0xc00af78840}, 0xc009c81700)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/handler.go:146 +0x5c6
k8s.io/apiserver/pkg/endpoints/filters.WithWebhookDuration.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/webhook_duration.go:31 +0x24e
net/http.HandlerFunc.ServeHTTP(0xc00b14c420, {0x4265d50, 0xc00af78840}, 0xc1678a)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
net/http.HandlerFunc.ServeHTTP(0xc00111a300, {0x4265d50, 0xc00af78840}, 0xc009772340)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/authorization.go:64 +0x41d
net/http.HandlerFunc.ServeHTTP(0x66785b8, {0x4265d50, 0xc00af78840}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
net/http.HandlerFunc.ServeHTTP(0x66079a0, {0x4265d50, 0xc00af78840}, 0x4)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/maxinflight.go:187 +0x29d
net/http.HandlerFunc.ServeHTTP(0xc00b14c420, {0x4265d50, 0xc00af78840}, 0xc1678a)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
net/http.HandlerFunc.ServeHTTP(0x44e712, {0x4265d50, 0xc00af78840}, 0x1)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/impersonation.go:50 +0x21c
net/http.HandlerFunc.ServeHTTP(0x66785b8, {0x4265d50, 0xc00af78840}, 0xa)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
net/http.HandlerFunc.ServeHTTP(0xc00b14c420, {0x4265d50, 0xc00af78840}, 0xc1678a)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
net/http.HandlerFunc.ServeHTTP(0x66785b8, {0x4265d50, 0xc00af78840}, 0xa)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
net/http.HandlerFunc.ServeHTTP(0xc00b14c420, {0x4265d50, 0xc00af78840}, 0xc1678a)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
net/http.HandlerFunc.ServeHTTP(0x427bc70, {0x4265d50, 0xc00af78840}, 0x3c75060)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x4265d50, 0xc00af78840}, 0xc00a064900)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/authentication.go:80 +0x839
net/http.HandlerFunc.ServeHTTP(0x427bc38, {0x4265d50, 0xc00af78840}, 0x41dc870)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4265d50, 0xc00af78840}, 0xc00a064700)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filterlatency/filterlatency.go:89 +0x46b
net/http.HandlerFunc.ServeHTTP(0xc009537380, {0x4265d50, 0xc00af78840}, 0x8e74f2)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/timeout.go:114 +0x70
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/timeout.go:100 +0x1dd

goroutine 709566 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1()
	/go/pkg/mod/golang.org/x/net@v0.0.0-20220412020605-290c469a71a5/http2/server.go:2177 +0x125
panic({0x32b8940, 0xc00b4c9d60})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc00121d088, 0x1, 0x17a0d66})
	/go/pkg/mod/k8s.io/apimachinery@v0.23.6/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x32b8940, 0xc00b4c9d60})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc00111a420, {0x4258e50, 0xc00064d990}, 0xdf8475800)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/timeout.go:120 +0x358
k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x4258e50, 0xc00064d990}, 0xc00a064500)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/request_deadline.go:100 +0x494
net/http.HandlerFunc.ServeHTTP(0x18, {0x4258e50, 0xc00064d990}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x4258e50, 0xc00064d990}, 0xc00a064500)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/waitgroup.go:77 +0x766
net/http.HandlerFunc.ServeHTTP(0x427bc70, {0x4258e50, 0xc00064d990}, 0x41dc870)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x4258e50, 0xc00064d990}, 0xc00a064400)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/warning.go:35 +0x2bb
net/http.HandlerFunc.ServeHTTP(0x373acc0, {0x4258e50, 0xc00064d990}, 0xd)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x4258e50, 0xc00064d990}, 0x3)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/cachecontrol.go:31 +0x126
net/http.HandlerFunc.ServeHTTP(0x203002, {0x4258e50, 0xc00064d990}, 0x30)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x4258e50, 0xc00064d990}, 0xc00a064400)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/httplog/httplog.go:108 +0xaf
net/http.HandlerFunc.ServeHTTP(0x427bc70, {0x4258e50, 0xc00064d990}, 0x41dc870)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x4258e50, 0xc00064d990}, 0xc00a064300)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/requestinfo.go:39 +0x316
net/http.HandlerFunc.ServeHTTP(0x427bc70, {0x4258e50, 0xc00064d990}, 0x2c)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x4258e50, 0xc00064d990}, 0xc00a064200)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/request_received_time.go:38 +0x27e
net/http.HandlerFunc.ServeHTTP(0x203002, {0x4258e50, 0xc00064d990}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x4258e50, 0xc00064d990}, 0xc00a064200)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x2c2
net/http.HandlerFunc.ServeHTTP(0x160, {0x4258e50, 0xc00064d990}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x4258e50, 0xc00064d990}, 0xc00a032f20)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/filters/wrap.go:74 +0xba
net/http.HandlerFunc.ServeHTTP(0x373acc0, {0x4258e50, 0xc00064d990}, 0x8)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withAuditID.func1({0x4258e50, 0xc00064d990}, 0xc009c81600)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/endpoints/filters/with_auditid.go:66 +0x40d
net/http.HandlerFunc.ServeHTTP(0xc000e55d60, {0x4258e50, 0xc00064d990}, 0xc000e55ccc)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc008641fa0, {0x4258e50, 0xc00064d990}, 0x427bc00)
	/go/pkg/mod/k8s.io/apiserver@v0.23.6/pkg/server/handler.go:189 +0x2b
net/http.serverHandler.ServeHTTP({0xc000e55d18}, {0x4258e50, 0xc00064d990}, 0xc009c81600)
	/usr/local/go/src/net/http/server.go:2879 +0x43b
net/http.initALPNRequest.ServeHTTP({{0x427bc70, 0xc0096623f0}, 0xc000b3c000, {0xc0001b27e0}}, {0x4258e50, 0xc00064d990}, 0xc009c81600)
	/usr/local/go/src/net/http/server.go:3480 +0x245
golang.org/x/net/http2.(*serverConn).runHandler(0xc000e55fd0, 0x8f288a, 0x0, 0xc000e55eb8)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20220412020605-290c469a71a5/http2/server.go:2184 +0x78
created by golang.org/x/net/http2.(*serverConn).processHeaders
	/go/pkg/mod/golang.org/x/net@v0.0.0-20220412020605-290c469a71a5/http2/server.go:1914 +0x590

keda-operator

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x29dfa56]

goroutine 1537 [running]:
github.com/kedacore/keda/v2/pkg/scalers.(*datadogScaler).getQueryResult(0xc013542c30, {0x3ae9008, 0xc01370aec0})
	/workspace/pkg/scalers/datadog_scaler.go:287 +0x696
github.com/kedacore/keda/v2/pkg/scalers.(*datadogScaler).IsActive(0x7, {0x3ae9008, 0xc01370aec0})
	/workspace/pkg/scalers/datadog_scaler.go:210 +0x25
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive(0xc0135008c0, {0x3ae9008, 0xc01370aec0}, 0xc012eb9e00)
	/workspace/pkg/scaling/cache/scalers_cache.go:88 +0xef
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers(0xc0001e44d0, {0x3ae9008, 0xc01370aec0}, {0x336f2c0, 0xc012eb9e00}, {0x3ac7030, 0xc013712c00})
	/workspace/pkg/scaling/scale_handler.go:278 +0x4b2
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop(0xc0001e44d0, {0x3ae9008, 0xc01370aec0}, 0xc012ecf180, {0x336f2c0, 0xc012eb9e00}, {0x3ac7030, 0xc013712c00})
	/workspace/pkg/scaling/scale_handler.go:149 +0x31c
created by github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).HandleScalableObject
	/workspace/pkg/scaling/scale_handler.go:105 +0x6ef

KEDA Version

2.7.1

Kubernetes Version

1.21

Platform

Amazon Web Services

Scaler Details

datadog

Anything else?

No response

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Roadmap - KEDA Core

  • Status

    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions