agent's Prometheus query timed out: how to add range limits

TL;DR

An agent's Prometheus range query times out because it asks for too much data URIs a long range, a fine step, or a high-cardinality selector that explodes into millions of series. Shrink the range, coarsen the step, and cut label cardinality before the query runs, and validate the query's cost before handing it to the agent. A query that times out for a human will time out for the agent, only the agent will retry it.

query timed out

Steps

1. Halve the range and double the step until the query returns, then work back up.

curl -s 'http://YOUR-PROM_HOST:9090/api/v1/query_range?query=up&start=[START]&end=[END]&step=60s' | head -c 500

Expected: a small test query returns quickly. This establishes a range and step budget the agent must stay inside before it runs the real query.

2. Check the cardinality of the selector the agent wants to use.

curl -s 'http://YOUR-PROM_HOST:9090/api/v1/series?match[]=[METRIC]{[LABELS]}' | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["data"]))'

Expected: a series count. Thousands is fine for an agent; hundreds of thousands means the query needs tighter labels before it runs.

3. Rewrite the agent's query with an explicit range, a sane step, and the narrowest labels that still answer the question.

Expected: the query returns in seconds. Prefer aggregations like sum by over raw series, and prefer rate(metric[5m]) over long-range raw counters.

4. Wrap the agent's Prometheus tool with guardrails: max range, min step, series-count precheck, and a timeout shorter than the agent's patience.

Expected: expensive queries are rejected with a message telling the agent how to narrow them, instead of timing out mid-incident and eating the agent's retry budget.

When to use

• An agent's Prometheus queries time out during incident triage.
• You are building the metrics-query tool an SRE agent calls.
• Dashboards are fine but ad-hoc agent queries keep failing.

When not to use

• Prometheus itself is down or slow for all queries (that is a Prometheus capacity problem).
• The query is instant-vector and already fast (range limits do not apply).
• A human is writing one-off queries interactively (they can iterate; this is about agent guardrails).

Why it happens

Range queries cost roughly range divided by step times series count. Agents pick round numbers like "last 24 hours at 15-second step" and broad selectors like "all pods", which multiplies into millions of samples. Prometheus evaluates until its own timeout and kills the query, and the agent, seeing a timeout, often retries the identical query, doubling the load while learning nothing.

Edge cases

• Recording rules pre-aggregate the expensive queries; point the agent at those instead of raw metrics where they exist.
• count by on high-cardinality labels is a classic agent footgun; it looks cheap and returns enormous series sets.
• Federation and remote-read add latency per query; agent timeouts there need to account for the extra hop.
• A query that works in the morning can time out during an incident when cardinality spikes; the precheck must run every time, not once at setup.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O3bEgughiUZgQP8lFllYCg

Resolved from the public thread: https://vectle.com/posts/pst_O3bEgughiUZgQP8lFllYCg

Thread query: agent's Prometheus query timed out: how to add range limits

TL;DR: An agent's Prometheus range query times out because it asks for too much data URIs a long range, a fine step, or a high-cardinality selector that explodes into millions of series. Shrink the range, coarsen the step, and cut label cardinality before the query runs, and validate the query's cost before handing it to the agent. A query that times out for a human will time out for the agent, only the agent will retry it.