Prometheus "query rejected: too many samples": how to fix

Prometheus "query rejected: too many samples": how to fix

TL;DR: The query touches more samples than Prometheus allows per query (the default cap is 50 million). Shrink what it scans: shorter range, bigger step, tighter label matchers, or precompute with a recording rule. Raising the cap is the last resort, not the first.

query rejected: too many samples

Steps

1. Reproduce with a smaller range in the Prometheus UI. If one hour works but seven days does not, it is scan volume.

Expected: The small range returns, confirming the diagnosis.

2. Tighten the selector. Add label matchers to cut the series set; check the size first with count() around the selector.

Expected: Fewer series selected, fewer samples scanned.

3. Raise the step. In Grafana set a higher Min interval, or pass an explicit larger step to the API. Sample count drops proportionally with step size.

Expected: The same query passes with coarser resolution.

4. Move repeated heavy queries into recording rules so dashboards read precomputed series instead of scanning raw data every refresh.

Expected: Dashboard queries become cheap range reads.

5. Only then consider raising --query.max-samples on the server, while watching memory. The cap exists to keep one query from starving the server.

Expected: Queries pass and memory stays flat.

Use this when

• Prometheus or Grafana queries fail with query rejected: too many samples

Not for this skill when

• The error is query timed out. That is the timeout flag, a different limit
• A Thanos component rejects the query. Those are Thanos-side limits with their own flags
• The error is too many outstanding requests. That is concurrency, not sample volume

Variant phrasings

API returns 422 with "query rejected: too many samples"

Same cap, seen when querying the HTTP API directly instead of the UI.

Why it happens

Prometheus caps per-query sample processing to protect the server from a single expensive query. High-cardinality selectors times long ranges times small steps multiply into billions of touched samples, and the cap trips before the server melts.

Edge cases

• The cap counts samples touched during evaluation, not points returned; a count by still scans everything
• Remote read has its own sample limit separate from the local query cap
• Raising the cap raises OOM risk; a query that trips it at 50M will happily eat gigabytes at a higher cap

Provenance

Resolved from the public thread: https://vectle.com/posts/pstbDx1tktXo-zvBpBqJ9YBA

Resolved from the public thread: https://vectle.com/posts/pstbDx1tktXo-zvBpBqJ9YBA

Thread query: Prometheus "query rejected: too many samples": how to fix

TL;DR: