Prometheus "query rejected: too many samples": how to fix
Fixes Prometheus queries rejected with 'query rejected: too many samples'. Use when a query scans more samples than the per-query cap allows; narrowing the range, raising the step, tightening label matchers, or precomputing with recording rules fixes it. Not for query timeouts, Thanos store limits, or request concurrency limits.
Prometheus "query rejected: too many samples": how to fix
TL;DR: The query touches more samples than Prometheus allows per query (the default cap is 50 million). Shrink what it scans: shorter range, bigger step, tighter label matchers, or precompute with a recording rule. Raising the cap is the last resort, not the first.
query rejected: too many samplesSteps
- Reproduce with a smaller range in the Prometheus UI. If one hour works but seven days does not, it is scan volume.
Expected: The small range returns, confirming the diagnosis.
- Tighten the selector. Add label matchers to cut the series set; check the size first with
count()around the selector.
Expected: Fewer series selected, fewer samples scanned.
- Raise the step. In Grafana set a higher Min interval, or pass an explicit larger step to the API. Sample count drops proportionally with step size.
Expected: The same query passes with coarser resolution.
- Move repeated heavy queries into recording rules so dashboards read precomputed series instead of scanning raw data every refresh.
Expected: Dashboard queries become cheap range reads.
- Only then consider raising
--query.max-sampleson the server, while watching memory. The cap exists to keep one query from starving the server.
Expected: Queries pass and memory stays flat.
Use this when
- Prometheus or Grafana queries fail with
query rejected: too many samples
Not for this skill when
- The error is
query timed out. That is the timeout flag, a different limit - A Thanos component rejects the query. Those are Thanos-side limits with their own flags
- The error is
too many outstanding requests. That is concurrency, not sample volume
Variant phrasings
API returns 422 with "query rejected: too many samples"
Same cap, seen when querying the HTTP API directly instead of the UI.
Why it happens
Prometheus caps per-query sample processing to protect the server from a single expensive query. High-cardinality selectors times long ranges times small steps multiply into billions of touched samples, and the cap trips before the server melts.
Edge cases
- The cap counts samples touched during evaluation, not points returned; a
count bystill scans everything - Remote read has its own sample limit separate from the local query cap
- Raising the cap raises OOM risk; a query that trips it at 50M will happily eat gigabytes at a higher cap
Provenance
Resolved from the public thread: https://vectle.com/posts/pstbDx1tktXo-zvBpBqJ9YBA