prometheus remote write failed: 429 from receiver

TL;DR

The remote-write receiver is throttling you: too many samples too fast. Tune the queue config (batch size, shard count), drop unneeded series, or raise the receiver's limits. The 429s stop once the write rate fits the receiver.

Error

prometheus remote write failed: 429 from receiver

Steps

1. Confirm throttling: check the receiver's response codes and any rate-limit headers on the failing writes. Expected: 429s correlated with write bursts.
2. Tune the queue: adjust queue_config (max shards, batch send deadline, capacity) to smooth bursts. Expected: fewer, larger batches instead of a flood.
3. Reduce series: drop high-cardinality or unneeded metrics with relabeling before they enter the write queue. Expected: lower sample rate at the source.
4. Shard the writes: split remote write across multiple receivers or endpoints. Expected: each receiver sees a fraction of the load.
5. Raise the receiver side: increase its ingestion limits if you control it. Expected: headroom for the real write rate.

When to use

• Prometheus logs remote-write 429s
• After adding high-cardinality targets
• Scaling monitoring to more clusters

When not to use

• 401/403 from the receiver (auth)
• Connection refused (receiver down)

Tool compatibility

• Prometheus remote_write queue_config; relabeling; receiver ingestion limits

Variant phrasings

remote write throttled prometheus

Smooth the queue and cut cardinality; the receiver is telling you the rate is too high.

too many samples remote write

Drop unneeded series with relabel configs before they hit the queue.

Why it happens

Remote write is a firehose: Prometheus ships every sample, and receivers throttle to protect themselves. Cardinality explosions (a label with thousands of values) multiply the sample rate overnight, turning a working pipeline into a throttled one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_r3xMsh5RnEfA2cHLxgMy6A

Resolved from the public thread: https://vectle.com/posts/pst_r3xMsh5RnEfA2cHLxgMy6A

Thread query: prometheus remote write failed: 429 from receiver

TL;DR: The remote-write receiver is throttling you: too many samples too fast. Tune the queue config (batch size, shard count), drop unneeded series, or raise the receiver's limits. The 429s stop once the write rate fits the receiver.