## TL;DR
A timing-out query is almost always a query over too much data URIs a huge time range, a high-cardinality aggregation, or a subquery explosion. Find it in the query log (slow query logging), look at its shape, then fix the shape: narrower ranges, pre-aggregated recording rules, fewer label dimensions. Do not raise the timeout; fix the query.

## The query
```text
Prometheus "query timed out": how to find the expensive query
```

## Use this when
- Dashboard panels or API queries time out
- You need to identify the slow query
- After adding new dashboards or alerts
- Query performance degrades over time

## Not for when
- Federation or remote read timeouts
- Storage performance issues
- Alert evaluation slowness (related but separate)

## Steps

### Step 1: Enable and read the slow query log
Turn on query logging with a slow threshold and find the queries exceeding it. The log gives you the exact query text and duration; guessing from dashboard behavior wastes time.
Expected output: the offending query text with its execution time.

### Step 2: Analyze the query shape
Look for the expensive patterns: aggregations over high-cardinality labels, long range vectors (rate over hours on raw data), subqueries, and regex matchers over huge series sets. One of these is always present.
Expected output: the expensive construct identified.

### Step 3: Narrow the query
Reduce the time range, add more specific label matchers, and drop unneeded dimensions from the aggregation. A query that scans 30 days usually only needs 1; a query grouping by pod usually only needs the service.
Expected output: the same answer from a fraction of the data.

### Step 4: Pre-aggregate with recording rules
For queries that must stay expensive (SLO dashboards, frequently viewed panels), compute them as recording rules on a schedule. The dashboard then queries the precomputed series, which is instant.
Expected output: the dashboard fast via recording rules; the raw query retired.

### Step 5: Set query guardrails
Configure max query duration, max samples, and per-query timeouts so one bad query cannot wedge the server. Expensive queries should fail fast and visibly, not hang the whole UI.
Expected output: bounded query behavior; the next bad query errors instead of hanging.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-WgcuOLtAQe81Ea1aM8TGQ