Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Native histograms: do not remote write exemplar if native histogram series is not sent #13552

Open
krajorama opened this issue Feb 7, 2024 · 2 comments
Open

Native histograms: do not remote write exemplar if native histogram series is not sent #13552

krajorama opened this issue Feb 7, 2024 · 2 comments

Comments

@krajorama
Copy link
Member

krajorama commented Feb 7, 2024 •
edited
Loading

What did you do?

Set up Prometheus with native histograms feature enabled, scraping classic histograms as well, but disable sending native histograms to a remote storage.

What did you expect to see?

No errors on remote write.

What did you see instead? Under which circumstances?

Remote (Mimir) reporting:

ts=2024-02-07T09:30:14.28217803Z caller=grpc_logging.go:66 level=warn method=/httpgrpc.HTTP/Handle duration=19.66365ms msg=gRPC err="rpc error: code = Code(400) desc = failed pushing to ingester ingester-zone-c-0: user=*****: the exemplar has been rejected because the related series has not been ingested yet (err-mimir-exemplar-series-missing). The affected exemplar is {traceID=\"715a8ef985d1ccba\"} with timestamp 2024-02-07T09:30:09.794Z for series cortex_request_duration_seconds{cluster=\"...\", container=\"query-scheduler\", instance=\"query-scheduler-5c68646cbf-gnw7m:query-scheduler:http-metrics\", job=\".../query-scheduler\", method=\"GET\", namespace=\"...\", pod=\"query-scheduler-5c68646cbf-gnw7m\", prometheus_job=\"migrate\", route=\"debug_pprof\", status_code=\"200\", ws=\"false\"}\n"

Note that the series cortex_request_duration_seconds is a native histogram series and I disabled writing native histogram series to the remote, so this is totally correct error from Mimir's point of view.

System information

Kubernetes, amd64

Prometheus version

prometheus, version 2.49.1 (branch: HEAD, revision: 43e14844a33b65e2a396e3944272af8b3a494071)
  build user:       root@6d5f4c649d25
  build date:       20240115-16:58:43
  go version:       go1.21.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file

global:
        scrape_interval: 15s
    remote_write:
        - headers:
            X-Scope-OrgId: "**********"
          send_exemplars: true
          send_native_histograms: false
          url: http://***********.svc.cluster.local./api/prom/push
    scrape_configs:
        - job_name: histograms
          kubernetes_sd_configs:
            - role: pod
          metric_relabel_configs:
            - action: keep
              regex: cortex_request_duration_seconds.*
              source_labels:
                - __name__
          relabel_configs:
            ....
          scrape_classic_histograms: true

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

ts=2024-02-07T10:11:49.283Z caller=dedupe.go:112 component=remote level=error remote_name=cd87f5 url=http://*********.svc.cluster.local./api/prom/push msg="non-recoverable error" count=1986 exemplarCount=14 err="server returned HTTP status 400 Bad Request: failed pushing to ingester ingester-zone-c-0: user=*****: the exemplar has been rejected because the related series has not been ingested yet (err-mimir-exemplar-series-missing). The affected exemplar is {traceID=\"66a9f18bba5436a9\"} with timestamp 2024-02-07T10:11:33.574Z for series cortex_request_duration_seconds{cluster=\"*****\", container=\"store-gateway\", instance=\"store-gateway-zone-a-0:store-gateway:http-metrics\", job=\"******/store-gateway-zone-a\", method=\"GET\", namespace=\"********\", pod=\"store-gateway-zone-a-0\", prometheus_job=\"migrate\", route=\"debug_pprof\", status_code=\"200\", ws=\"false\"}"
@krajorama
Copy link
Member Author

Not high priority.

@krajorama
Copy link
Member Author

Also it seems like remote write 2.0 will solve this by sending series and exemplars bundled together. Ref: #13105 . Should be possible to close after testing with remote write 2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@krajorama