# Prometheus "query rejected: too many samples": how to fix

TL;DR: The query touches more samples than Prometheus allows per query (the default cap is 50 million). Shrink what it scans: shorter range, bigger step, tighter label matchers, or precompute with a recording rule. Raising the cap is the last resort, not the first.

```text
query rejected: too many samples
```

## Steps

1. Reproduce with a smaller range in the Prometheus UI. If one hour works but seven days does not, it is scan volume.
**Expected:** The small range returns, confirming the diagnosis.

2. Tighten the selector. Add label matchers to cut the series set; check the size first with `count()` around the selector.
**Expected:** Fewer series selected, fewer samples scanned.

3. Raise the step. In Grafana set a higher Min interval, or pass an explicit larger step to the API. Sample count drops proportionally with step size.
**Expected:** The same query passes with coarser resolution.

4. Move repeated heavy queries into recording rules so dashboards read precomputed series instead of scanning raw data every refresh.
**Expected:** Dashboard queries become cheap range reads.

5. Only then consider raising `--query.max-samples` on the server, while watching memory. The cap exists to keep one query from starving the server.
**Expected:** Queries pass and memory stays flat.

## Use this when
- Prometheus or Grafana queries fail with `query rejected: too many samples`

## Not for this skill when
- The error is `query timed out`. That is the timeout flag, a different limit
- A Thanos component rejects the query. Those are Thanos-side limits with their own flags
- The error is `too many outstanding requests`. That is concurrency, not sample volume

## Variant phrasings
### API returns 422 with "query rejected: too many samples"
Same cap, seen when querying the HTTP API directly instead of the UI.

## Why it happens
Prometheus caps per-query sample processing to protect the server from a single expensive query. High-cardinality selectors times long ranges times small steps multiply into billions of touched samples, and the cap trips before the server melts.

## Edge cases
- The cap counts samples touched during evaluation, not points returned; a `count by` still scans everything
- Remote read has its own sample limit separate from the local query cap
- Raising the cap raises OOM risk; a query that trips it at 50M will happily eat gigabytes at a higher cap

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bDx1tktXo-zvBpB_qJ9YBA