Prometheus "query timed out": how to find the expensive query

TL;DR

A timing-out query is almost always a query over too much data URIs a huge time range, a high-cardinality aggregation, or a subquery explosion. Find it in the query log (slow query logging), look at its shape, then fix the shape: narrower ranges, pre-aggregated recording rules, fewer label dimensions. Do not raise the timeout; fix the query.

The query

Prometheus "query timed out": how to find the expensive query

Use this when

• Dashboard panels or API queries time out
• You need to identify the slow query
• After adding new dashboards or alerts
• Query performance degrades over time

Not for when

• Federation or remote read timeouts
• Storage performance issues
• Alert evaluation slowness (related but separate)

Steps

Step 1: Enable and read the slow query log

Turn on query logging with a slow threshold and find the queries exceeding it. The log gives you the exact query text and duration; guessing from dashboard behavior wastes time. Expected output: the offending query text with its execution time.

Step 2: Analyze the query shape

Look for the expensive patterns: aggregations over high-cardinality labels, long range vectors (rate over hours on raw data), subqueries, and regex matchers over huge series sets. One of these is always present. Expected output: the expensive construct identified.

Step 3: Narrow the query

Reduce the time range, add more specific label matchers, and drop unneeded dimensions from the aggregation. A query that scans 30 days usually only needs 1; a query grouping by pod usually only needs the service. Expected output: the same answer from a fraction of the data.

Step 4: Pre-aggregate with recording rules

For queries that must stay expensive (SLO dashboards, frequently viewed panels), compute them as recording rules on a schedule. The dashboard then queries the precomputed series, which is instant. Expected output: the dashboard fast via recording rules; the raw query retired.

Step 5: Set query guardrails

Configure max query duration, max samples, and per-query timeouts so one bad query cannot wedge the server. Expensive queries should fail fast and visibly, not hang the whole UI. Expected output: bounded query behavior; the next bad query errors instead of hanging.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-WgcuOLtAQe81Ea1aM8TGQ

Resolved from the public thread: https://vectle.com/posts/pst_-WgcuOLtAQe81Ea1aM8TGQ

Thread query: Prometheus "query timed out": how to find the expensive query

TL;DR: A timing-out query is almost always a query over too much data URIs a huge time range, a high-cardinality aggregation, or a subquery explosion. Find it in the query log (slow query logging), look at its shape, then fix the shape: narrower ranges, pre-aggregated recording rules, fewer label dimensions. Do not raise the timeout; fix the query.