Skip to content

jaeger-query: "msg":"Failed to exec query","error":"Operation failed - received 0 responses and 1 failures" - wut? #418

New issue
New issue
Closed
Closed
jaeger-query: "msg":"Failed to exec query","error":"Operation failed - received 0 responses and 1 failures" - wut?#418
@Dieterbe

Description

@Dieterbe
Dieterbe
opened on Sep 22, 2017

sometimes when i search from the UI I get:

There was an error querying for traces:
Bad JSON returned from the Jaeger Query Service.

according to network tab in devtools of the browser, the json response was:

{"data":null,"total":0,"limit":0,"offset":0,"errors":[{"code":500,"msg":"Operation failed - received 0 responses and 1 failures"}]}

and the pod logs show:

{"level":"error","ts":1506096484.0922747,"caller":"spanstore/reader.go:376","msg":"Failed to exec query","error":"Operation failed - received 0 responses and 1 failures","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:176\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).executeQuery\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:376\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).queryByService\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:362\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).findTraceIDs\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:280\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).FindTraces\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:236\ngithub.com/uber/jaeger/cmd/query/app.(*APIHandler).search\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/app/handler.go:212\ngithub.com/uber/jaeger/cmd/query/app.(*APIHandler).(github.com/uber/jaeger/cmd/query/app.search)-fm\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/app/handler.go:124\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1726\ngithub.com/uber/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1726\nnet/http.(Handler).ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/h2_bundle.go:4084\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1726\ngithub.com/uber/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/gorilla/mux/mux.go:114\ngithub.com/uber/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/uber/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:53\nnet/http.serverHandler.ServeHTTP\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:2202\nnet/http.(*conn).serve\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1579"}

so the problem is:

  1. error in UI is not really informational about what the problem is and where to look next (in particular, the returned json looks fine to me)
  2. the error in the log doesn't say much either. what operations failed, what's the failure? where do i look next? is there other services involved?

PS: sorry for just opening a ticket and not providing a PR. i would like to contribute more eventually but right now i'm quite busy with work.

Activity

tiffon
mentioned this on Nov 27, 2017
tiffon

tiffon commented on Nov 27, 2017

@tiffon
tiffon
on Nov 27, 2017
Member
  1. error in UI is not really informational about what the problem is and where to look next (in particular, the returned json looks fine to me)

@Dieterbe I created jaegertracing/jaeger-ui#127 to address the first problem you mention.

Thanks for reporting this.

black-adder
added
ui
on Nov 28, 2017
tiffon
mentioned this on Dec 14, 2017
tiffon

tiffon commented on Dec 19, 2017

@tiffon
tiffon
on Dec 19, 2017 · edited by tiffon
Member
  1. error in UI is not really informational about what the problem is and where to look next (in particular, the returned json looks fine to me)

1 was fixed by jaegertracing/jaeger-ui#133.

yurishkuro

yurishkuro commented on Dec 23, 2017

@yurishkuro
yurishkuro
on Dec 23, 2017
Member

"error":"Operation failed - received 0 responses and 1 failures"

Not sure how we can do better, it seems to be the limitation of the gocql driver, because we do log the error returned from it:

	for i.Scan(&traceID) {
		retMe.Add(traceID)
	}
	err := i.Close()
	if err != nil {
		s.logger.Error("Failed to exec query", zap.Error(err))
		return nil, err
	}
yurishkuro
removed
ui
on Dec 23, 2017
jpkrohling

jpkrohling commented on Apr 2, 2019

@jpkrohling
jpkrohling
on Apr 2, 2019
Contributor

Closing this, as the UI part was done already and the backend part won't be done, as far as I understood.

jpkrohling
closed this as completedon Apr 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      jaeger-query: "msg":"Failed to exec query","error":"Operation failed - received 0 responses and 1 failures" - wut? · Issue #418 · jaegertracing/jaeger