Skip to content

Is argument -validation.reject-old-samples.max-age valid for distributor or ingester or both? #6220

Open
@mousimin

Description

Describe the bug
I am using micro services mode for cortex, for distributor, I am using -validation.reject-old-samples & -validation.reject-old-samples.max-age arguments, but for ingester, I am not using them.

  -validation.reject-old-samples=true \
  -validation.reject-old-samples.max-age=28d

I am using VictoriaMetrics to remote write the metrics to cortex, I got an "too old sample" error which I guess comes from ingester:
{"ts":"2024-09-18T03:21:11.282Z","level":"error","caller":"VictoriaMetrics/app/vmagent/remotewrite/client.go:400","msg":"sending a block with size 1680 bytes to \"1:secret-url\" was rejected (skipping the block): status code 400; response body: maxFailure (quorum) on a given error family, rpc error: code = Code(400) desc = addr=15.132.24.168:2012 state=ACTIVE zone=zone1, rpc error: code = Code(400) desc = user=user1: err: too old sample. timestamp=2024-09-13T09:08:53.521Z, series={...}"}

Checked the code, I think distributor will validate the max-age value:
https://github.com/cortexproject/cortex/blob/master/pkg/distributor/distributor.go#L557

But for ingester, I know it will mainly reuse the code of Prometheus, but I didn't find any code which will do the same validation for distributor, for instead I found there is a argument -ingester.out-of-order-time-window which I specified for ingester with value 6h.

Should I need to add arguments -validation.reject-old-samples.max-age for ingester as well? or should I use the same value for -ingester.out-of-order-time-window with -validation.reject-old-samples.max-age? or this is a bug?

To Reproduce
Steps to reproduce the behavior:

  1. Start Cortex v1.17.1 with micro services mode, configure -validation.reject-old-samples & -validation.reject-old-samples.max-age only for distributor
  2. Start a VictoriaMetrics or Prometheus to remote write some old metrics.

Expected behavior
No error should appear

Environment:

  • Infrastructure: bare-metal
  • Deployment tool: ansible

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions