Loki log queries timing out: how to speed them up

TL;DR

Loki is fast when queries are narrow (few streams, short time range) and slow when they are wide (many streams, long ranges, heavy regex). Speed it up by querying with exact label matchers first, keeping time ranges tight, and pushing filtering into label matchers instead of line filters. Most timeouts are queries that scan too much.

The query

Loki log queries timing out: how to speed them up

Use this when

• LogQL queries time out or take minutes
• Loki-backed dashboard panels are slow
• You are designing label schemas for Loki
• Choosing between Loki and other log stores

Not for when

• Prometheus query performance
• Elasticsearch tuning
• Log ingestion or agent configuration

Steps

Step 1: Narrow with label matchers first

Rewrite the query to select the smallest stream set with exact label matchers before any line filters. Loki's speed comes from stream selection; a query over all streams with a regex filter is the slowest possible shape. Expected output: the same results returned dramatically faster, or a clear stream-selection improvement.

Step 2: Shorten the time range

Cut the query window to the minimum that answers the question. Debugging the last hour does not need a 7-day query. Long ranges multiply the data scanned linearly. Expected output: queries complete in seconds instead of timing out.

Step 3: Prefer label filters over line filters

Move conditions from line filters (|=, |~) into label matchers wherever the data is labeled. Line filters scan every log line in the selected streams; label matchers skip whole streams. Expected output: fewer lines scanned per query; regex filters reserved for what labels cannot express.

Step 4: Check label cardinality

Too many streams (high label cardinality) slows everything: each stream is overhead. If one label has thousands of values, consider whether it belongs in the log line instead of as a label. Loki labels should be low-cardinality by design. Expected output: a label schema with bounded stream counts; query planning overhead reduced.

Step 5: Split and parallelize big analytical queries

For unavoidable wide queries (weekly reports, incident forensics), split by time into smaller queries run in parallel, or use Loki's query sharding. Do not run one giant query and hope. Expected output: large analyses complete through parallel chunks instead of timing out as one query.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zjfjZPG8umoHZW7qGIwIGQ

Resolved from the public thread: https://vectle.com/posts/pst_zjfjZPG8umoHZW7qGIwIGQ

Thread query: Loki log queries timing out: how to speed them up

TL;DR: Loki is fast when queries are narrow (few streams, short time range) and slow when they are wide (many streams, long ranges, heavy regex). Speed it up by querying with exact label matchers first, keeping time ranges tight, and pushing filtering into label matchers instead of line filters. Most timeouts are queries that scan too much.