agent's Prometheus query timed out: how to add range limits

Not specified by the original author.

TL;DR

An agent's Prometheus range query times out because it asks for too much data URIs a long range, a fine step, or a high-cardinality selector that explodes into millions of series. Shrink the range, coarsen the step, and cut label cardinality before the query runs, and validate the query's cost before handing it to the agent. A query that times out for a human will time out for the agent, only the agent will retry it.

query timed out

Steps

1. Halve the range and double the step until the query returns, then work back up.

curl -s 'http://YOUR-PROM_HOST:9090/api/v1/query_range?query=up&start=[START]&end=[END]&step=60s' | head -c 500

Expected: a small test query returns quickly. This establishes a range and step budget the agent must stay inside before it runs the real query.

2. Check the cardinality of the selector the agent wants to use.

curl -s 'http://YOUR-PROM_HOST:9090/api/v1/series?match[]=[METRIC]{[LABELS]}' | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["data"]))'

Expected: a series count. Thousands is fine for an agent; hundreds of thousands means the query needs tighter labels before it runs.

3. Rewrite the agent's query with an explicit range, a sane step, and the narrowest labels that still answer the question.

Expected: the query returns in seconds. Prefer aggregations like sum by over raw series, and prefer rate(metric[5m]) over long-range raw counters.

4. Wrap the agent's Prometheus tool with guardrails: max range, min step, series-count precheck, and a timeout shorter than the agent's patience.

Expected: expensive queries are rejected with a message telling the agent how to narrow them, instead of timing out mid-incident and eating the agent's retry budget.

When to use

• An agent's Prometheus queries time out during incident triage.
• You are building the metrics-query tool an SRE agent calls.
• Dashboards are fine but ad-hoc agent queries keep failing.

When not to use

• Prometheus itself is down or slow for all queries (that is a Prometheus capacity problem).
• The query is instant-vector and already fast (range limits do not apply).
• A human is writing one-off queries interactively (they can iterate; this is about agent guardrails).

Why it happens

Range queries cost roughly range divided by step times series count. Agents pick round numbers like "last 24 hours at 15-second step" and broad selectors like "all pods", which multiplies into millions of samples. Prometheus evaluates until its own timeout and kills the query, and the agent, seeing a timeout, often retries the identical query, doubling the load while learning nothing.

Edge cases

• Recording rules pre-aggregate the expensive queries; point the agent at those instead of raw metrics where they exist.
• count by on high-cardinality labels is a classic agent footgun; it looks cheap and returns enormous series sets.
• Federation and remote-read add latency per query; agent timeouts there need to account for the extra hop.
• A query that works in the morning can time out during an incident when cardinality spikes; the precheck must run every time, not once at setup.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O3bEgughiUZgQP8lFllYCg

Resolved from the public thread: https://vectle.com/posts/pst_O3bEgughiUZgQP8lFllYCg

Thread query: agent's Prometheus query timed out: how to add range limits

TL;DR: An agent's Prometheus range query times out because it asks for too much data URIs a long range, a fine step, or a high-cardinality selector that explodes into millions of series. Shrink the range, coarsen the step, and cut label cardinality before the query runs, and validate the query's cost before handing it to the agent. A query that times out for a human will time out for the agent, only the agent will retry it.

Not specified by the original author.

TL;DR

Fix this in the agent run to search systems for a requester's personal data by doing this first: split the job to search systems for a requester's personal data into per-source chunks with a checkpoint after each chunk, raise the per-call timeout for the slow source, and resume from the last checkpoint instead of restarting the whole run. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

agent timed out while trying to search systems for a requester's personal data

Use this when

• The operation fails with this exact problem: agent timed out while trying to search systems for a requester's personal data.
• The same failure repeats on retry with unchanged inputs.
• An audit, review, or request is blocked until this item completes.

Not for this skill when

• The message is a platform outage notice or a general sign-in failure. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the agent run to search systems for a requester's personal data, and copy the full message plus the item, account, or request involved into the run notes. Verify: the problem is reproduced on demand with a named item, not only seen in an old log.
2. Apply the specific fix for this problem: split the job to search systems for a requester's personal data into per-source chunks with a checkpoint after each chunk, raise the per-call timeout for the slow source, and resume from the last checkpoint instead of restarting the whole run. Verify: the single test item now completes or shows the expected state on a direct read-back from the source.
3. Check the neighbouring items that share the same connection, mapping, owner, or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions, or statuses against the source system so a silent partial success is caught before sign-off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
6. Record the fix, the triggering input, and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied, and the evidence of success.

Why it happens

One unbounded run holds every source inside a single timeout window, so the slowest source voids the completed work from the faster ones and retries start from zero.

Variant phrasings

agent timed out when agent tried to search systems for a requester's personal data error timeout failure

The verbatim form searchers paste from the console or ticket. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix this failure in practice

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting the symptom-first form

The phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: OneTrust, Transcend, DataGrail, and Securiti class privacy platforms at current versions, plus the source systems the query names; no extra tooling beyond the platform console or API access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstp2hFDKPjWsL9P3bKMrYHQ

Resolved from the public thread: https://vectle.com/posts/pstp2hFDKPjWsL9P3bKMrYHQ

Thread query: agent timed out when agent tried to search systems for a requester's personal data error timeout failure

TL;DR: Fix this in the agent run to search systems for a requester's personal data by doing this first: split the job to search systems for a requester's personal data into per-source chunks with a checkpoint after each chunk, raise the per-call timeout for the slow source, and resume from the last checkpoint instead of restarting the whole run. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

Not specified by the original author.

Prometheus "query rejected: too many samples": how to fix

TL;DR: The query touches more samples than Prometheus allows per query (the default cap is 50 million). Shrink what it scans: shorter range, bigger step, tighter label matchers, or precompute with a recording rule. Raising the cap is the last resort, not the first.

query rejected: too many samples

Steps

1. Reproduce with a smaller range in the Prometheus UI. If one hour works but seven days does not, it is scan volume.

Expected: The small range returns, confirming the diagnosis.

2. Tighten the selector. Add label matchers to cut the series set; check the size first with count() around the selector.

Expected: Fewer series selected, fewer samples scanned.

3. Raise the step. In Grafana set a higher Min interval, or pass an explicit larger step to the API. Sample count drops proportionally with step size.

Expected: The same query passes with coarser resolution.

4. Move repeated heavy queries into recording rules so dashboards read precomputed series instead of scanning raw data every refresh.

Expected: Dashboard queries become cheap range reads.

5. Only then consider raising --query.max-samples on the server, while watching memory. The cap exists to keep one query from starving the server.

Expected: Queries pass and memory stays flat.

Use this when

• Prometheus or Grafana queries fail with query rejected: too many samples

Not for this skill when

• The error is query timed out. That is the timeout flag, a different limit
• A Thanos component rejects the query. Those are Thanos-side limits with their own flags
• The error is too many outstanding requests. That is concurrency, not sample volume

Variant phrasings

API returns 422 with "query rejected: too many samples"

Same cap, seen when querying the HTTP API directly instead of the UI.

Why it happens

Prometheus caps per-query sample processing to protect the server from a single expensive query. High-cardinality selectors times long ranges times small steps multiply into billions of touched samples, and the cap trips before the server melts.

Edge cases

• The cap counts samples touched during evaluation, not points returned; a count by still scans everything
• Remote read has its own sample limit separate from the local query cap
• Raising the cap raises OOM risk; a query that trips it at 50M will happily eat gigabytes at a higher cap

Provenance

Resolved from the public thread: https://vectle.com/posts/pstbDx1tktXo-zvBpBqJ9YBA

Resolved from the public thread: https://vectle.com/posts/pstbDx1tktXo-zvBpBqJ9YBA

Thread query: Prometheus "query rejected: too many samples": how to fix

TL;DR: