Closed
Description
Report
While using Keda and the ActiveMQ scaler, I found that I am unable to connect to ActiveMQ version 5.17.0 out of the box. The error that I get is ERROR activeMQ_scaler Unable to access activeMQ management endpoint {"managementEndpoint": "pobox.xphyrlab.net:8161", "error": "ActiveMQ management endpoint response error code : 200 403"} github.com/kedacore/keda/v2/pkg/scalers.(*activeMQScaler).IsActive I have also tested version 5.16.4 of ActiveMQ and get the same results. This appears to obe related to the "CORS" configuration of Jolokia blocking the Keda application due to an Origin header that is "NULL".
The work-around is to edit the <apache-activemq-root>/webapps/api/WEB-INF/classes/jolokia-access.xml file and remove the following lines:
<!-- Enforce that an Origin/Referer header is present to prevent CSRF -->
<cors>
<strict-checking/>
</cors>The problem goes away.
Expected Behavior
I expected that the ActiveMQ scaler would work out of the box, without needing to re-configure or remove default ActiveMQ configuration.
Actual Behavior
Keda scaller is unable to connect to ActiveMQ to monitor queue depth and thus is unable to scale target workloads.
Steps to Reproduce the Problem
- Install ActiveMQ from https://activemq.apache.org/components/classic/download/ using defaults
- Create a Keda ScaledObject that points to the ActiveMQ instance
- Check the logs of the "keda-operator" and "keda-metrics-apiserver" for the error logs
- edit
<apache-activemq-root>/webapps/api/WEB-INF/classes/jolokia-access.xmland remove the CORS configuration and restart ActiveMQ - Review the Keda logs and see that the connection is now working
Logs from KEDA operator
keda-operator logs
.6492533349499571e+09 ERROR activeMQ_scaler Unable to access activeMQ management endpoint {"managementEndpoint": "pobox.xphyrlab.net:8161", "error": "ActiveMQ management endpoint response error code : 200 403"}
github.com/kedacore/keda/v2/pkg/scalers.(*activeMQScaler).IsActive
/workspace/pkg/scalers/activemq_scaler.go:149
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
/workspace/pkg/scaling/cache/scalers_cache.go:87
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:149
1.649253334966924e+09 ERROR activeMQ_scaler Unable to access activeMQ management endpoint {"managementEndpoint": "pobox.xphyrlab.net:8161", "error": "ActiveMQ management endpoint response error code : 200 403"}
github.com/kedacore/keda/v2/pkg/scalers.(*activeMQScaler).IsActive
/workspace/pkg/scalers/activemq_scaler.go:149
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
/workspace/pkg/scaling/cache/scalers_cache.go:92
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:149
1.6492533349671416e+09 ERROR scalehandler Error getting scale decision {"scaledobject.Name": "activemq-scaledobject", "scaledObject.Namespace": "scaller", "scaleTarget.Name": "amqpexample-consumer", "error": "ActiveMQ management endpoint response error code : 200 403"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:278
keda-metrics-apiserver
E0406 13:55:24.511506 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
E0406 13:55:39.988464 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error inspecting ActiveMQ queue size: ActiveMQ management endpoint response error code : 200 403" "scaledObject.Name"="activemq-scaledobject" "scaledObject.Namespace"="scaller" "scaler"={}
E0406 13:55:39.988649 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
E0406 13:55:55.124037 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error inspecting ActiveMQ queue size: ActiveMQ management endpoint response error code : 200 403" "scaledObject.Name"="activemq-scaledobject" "scaledObject.Namespace"="scaller" "scaler"={}
E0406 13:55:55.124135 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
E0406 13:56:10.413379 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error inspecting ActiveMQ queue size: ActiveMQ management endpoint response error code : 200 403" "scaledObject.Name"="activemq-scaledobject" "scaledObject.Namespace"="scaller" "scaler"={}
E0406 13:56:10.413524 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
E0406 13:56:25.689172 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error inspecting ActiveMQ queue size: ActiveMQ management endpoint response error code : 200 403" "scaledObject.Name"="activemq-scaledobject" "scaledObject.Namespace"="scaller" "scaler"={}
E0406 13:56:25.689410 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"no matching metrics found for s0-activemq-keda-test"}: no matching metrics found for s0-activemq-keda-test
KEDA Version
2.6.1
Kubernetes Version
1.23
Platform
Red Hat OpenShift
Scaler Details
ActiveMQ
Anything else?
Using tcpdump I captured the traffic from the Keda application to ActiveMQ. The error that is returned from ActiveMQ is:
HTTP/1.1 200 OK
Date: Wed, 06 Apr 2022 13:03:02 GMT
X-FRAME-OPTIONS: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Type: text/plain;charset=utf-8
Cache-Control: no-cache
Pragma: no-cache
Expires: Wed, 06 Apr 2022 12:03:02 GMT
Transfer-Encoding: chunked
7F
{"error_type":"java.lang.Exception","error":"java.lang.Exception : Origin null is not allowed to call this agent","status":403}
I think I know a long term fix for this, and will create a followup PR that hopefully fixes this issue.
Metadata
Assignees
Labels
Type
Projects
Status
Done