Prometheus "out of order sample": what causes it and how to fix

TL;DR

Out-of-order samples happen when two writers send samples for the same series with overlapping timestamps: HA Prometheus pairs both remote-writing, restarted agents replaying old data, or clock skew making new samples look old. The fix is ensuring one writer per series per time range: deduplicate at the receiver, align external labels, and fix the clock. It is a pipeline topology problem, not a Prometheus bug.

The query

Prometheus "out of order sample": what causes it

Use this when

• Remote write receivers log out-of-order errors
• HA Prometheus pairs both write to one receiver
• Agents replay data after restarts
• Timestamps appear to go backwards

Not for when

• PromQL query errors
• Storage corruption
• Scrape failures

Steps

Step 1: Identify the duplicate writers

Find which two sources write the same series: HA pairs, overlapping federation, or an agent plus a server scraping the same target. The error names the series; the topology names the writers. Expected output: the duplicate writer pair identified.

Step 2: Deduplicate with external labels

Give each writer distinct external labels (replica identifiers) so their series do not collide, and deduplicate at query time. Identical external labels on two writers is the misconfiguration. Expected output: writers distinguishable; collisions eliminated.

Step 3: Check clock synchronization

Clock skew between writers makes one's samples look older than the other's. Verify NTP sync on all writers; even modest skew generates out-of-order errors at high scrape rates. Expected output: clocks in sync across writers.

Step 4: Handle agent replay after restarts

Agents that buffer and replay after downtime resend old samples the receiver already has. Configure replay windows sensibly and accept that some duplication at restart boundaries is normal; the receiver should tolerate it. Expected output: replay behavior bounded and understood.

Step 5: Monitor the error rate as a health signal

Track out-of-order error rates at the receiver. A steady low rate is HA noise; a spike means a topology change or clock problem. Alert on the spike, not the baseline. Expected output: the error rate baselined with alerts on real changes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DE2rPOLoH18QvNQ2NA6daA

Resolved from the public thread: https://vectle.com/posts/pst_DE2rPOLoH18QvNQ2NA6daA

Thread query: Prometheus "out of order sample": what causes it and how to fix

TL;DR: Out-of-order samples happen when two writers send samples for the same series with overlapping timestamps: HA Prometheus pairs both remote-writing, restarted agents replaying old data, or clock skew making new samples look old. The fix is ensuring one writer per series per time range: deduplicate at the receiver, align external labels, and fix the clock. It is a pipeline topology problem, not a Prometheus bug.