Loki "too many outstanding requests" 429: ingester tuning

TL;DR

The 429 means ingesters are saturated: more concurrent push requests than they can handle. The levers are ingester concurrency limits, the number of ingesters, and client-side backoff. Raising limits without adding capacity just moves the failure; scale ingesters or reduce push concurrency, and make sure clients back off properly instead of hammering.

The query

Loki "too many outstanding requests" 429: ingester tuning

Use this when

• Log shippers get 429s from Loki
• Ingestion falls behind during peaks
• Scaling the Loki write path
• After adding new log sources

Not for when

• Loki query timeouts (read path, different)
• Shipper configuration errors
• Storage backend issues

Steps

Step 1: Confirm ingester saturation, not a limit misconfig

Check ingester CPU, memory, and active request counts during the 429s. Saturated ingesters need capacity; idle ingesters returning 429s need limit tuning. Expected output: saturation confirmed or ruled out.

Step 2: Scale ingesters or raise per-ingester limits deliberately

Add ingester replicas for real saturation; tune max concurrent requests only when ingesters have headroom. Document which lever you pulled and why. Expected output: 429 rate dropping with the chosen change.

Step 3: Fix client push concurrency

Check shippers' batch sizes and concurrency: many small pushes overwhelm ingesters faster than fewer large ones. Bigger batches, fewer concurrent requests is the healthy shape. Expected output: push patterns reshaped to fewer, larger batches.

Step 4: Ensure clients back off on 429

Verify shippers honor Retry-After and back off exponentially. Clients that retry immediately turn a brief overload into a sustained one; backoff is what makes 429s self-healing. Expected output: 429s followed by backoff, not by retry storms.

Step 5: Alert on 429 rate, not just presence

Some 429s during extreme peaks are normal backpressure. Alert when the rate exceeds the baseline or when ingestion lag grows, not on single 429s. Expected output: alerts distinguishing healthy backpressure from real saturation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstqckDR6WAYtHm8JmxSvZrw