[Bug]: "HTTP handler, Internal Server Error","error":"search services failed.." #5381
Open
Description
What happened?
When want to get traces, there was 504 error code.
All worked fine before upgrade jaeger with the same version of Elastic Search.
Steps to reproduce
Just install the new version of Jaeger stack.
Expected behavior
Jaeger UI should work as expected.
jaeger collector config:
es:
version: 7
server-urls: "https://localhost:9200"
username: ""
password: ""
index-date-separator: "."
create-index-templates: false
bulk:
size: 8000
workers: 6
tags-as-fields:
all: true
tls:
enabled: true
skip-host-verify: true
jaeger query config:
log-level: debug
admin:
http:
host-port: 127.0.0.1:16687
query:
grpc-server:
host-port: 127.0.0.1:16685
http-server:
host-port: 127.0.0.1:16686
es:
server-urls: "https://localhost:9200"
username: ""
password: ""
index-date-separator: "."
tls:
enabled: true
skip-host-verify: true
bulk:
workers: 4
Relevant log output
jaeger-elk_deprecation.log
[CRITICAL][o.e.d.i.m.T.TypeFieldType] [jaeger-elk] [types removal] Using the _type field in queries and aggregations is deprecated, prefer to use a field instead.
jaeger-query[1952612]: {"level":"error","ts":1713781124.9002626,"caller":"app/http_handler.go:504","msg":"HTTP handler, Internal Server Error","error":"search services failed: Post \"https://localhost:9200/jaeger-span-2024.04.22/_search?ignore_unavailable=true&rest_total_hits_as_int=true\": context canceled","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:504\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:259\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:548\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:281\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:225\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:83\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.1/mux.go:212\ngithub.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6\n\tgithub.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:3137\nnet/http.(*conn).serve\n\tnet/http/server.go:2039"}Screenshot
Additional context
No response
Jaeger backend version
v1.56.0
SDK
No response
Pipeline
OTEL sdk -> OTEL collector -> Jaeger collector -> Elastic Search <- Jaeger Query -> Jaeger UI
Stogage backend
Elastic Search 7.17.14
Operating system
Linux
Deployment model
binaries on bare metal
Deployment configs
Ansible roles