Loki "entry out of order": timestamp debugging

Loki "entry out of order": timestamp debugging

TL;DR: Loki rejects any log line older than the newest line it already holds for that exact stream. That almost always means two writers are pushing the same stream, or a writer's clock jumped backward. Find and remove the duplicate writer first.

entry out of order

Steps

1. Identify the stream from the rejected batch. The error or the client log names the labels.

Expected: You know exactly which stream is rejecting.

2. Hunt for duplicate writers. Two collectors tailing the same file is the top cause: a DaemonSet plus a manually installed agent, overlapping file globs, or a sidecar duplicating the node's collector.

Expected: Every file path is tailed by exactly one collector.

3. Check clocks. Compare time on the writers and on Loki; NTP or chrony should be in sync everywhere. A writer whose clock runs behind (or jumped back after a reboot) produces stale timestamps.

Expected: Clocks agree within a second or two.

4. If some disorder is unavoidable, enable out-of-order ingestion in the limits config with a small window. The exact key depends on your Loki version, so check its docs for the current name.

Expected: Mildly disordered batches are accepted instead of rejected.

5. Check for retry storms. A push that keeps failing and retrying with old batches multiplies the rejects; fix the underlying push failure and the out-of-order noise usually stops with it.

Expected: Reject rate drops to near zero.

Use this when

• Loki push or distributor logs, or client errors, show entry out of order

Not for this skill when

• The error is too many labels. That is cardinality, a different limit
• The error is rate limiting on ingestion. That is volume, not ordering
• The error is a stream count limit. That is too many streams, not disorder

Variant phrasings

rpc error: code = Unknown desc = entry out of order

The gRPC-wrapped form of the same rejection, seen in distributor logs.

Why it happens

Loki stores each stream's chunks in timestamp order. An entry older than the stream's newest stored entry cannot be slotted into the already-written position, so it is rejected unless out-of-order ingestion is explicitly enabled.

Edge cases

• Out-of-order acceptance costs extra memory and CPU; keep the window small
• Duplicate lines from at-least-once delivery are normal and get dropped quietly; a few rejects are not a problem, a flood is
• Restarting an agent can re-send its buffer with old timestamps; that burst of rejects after a restart is expected and harmless

Provenance

Resolved from the public thread: https://vectle.com/posts/pstkFZ4VokKZyuRJ18R8cOkA

Resolved from the public thread: https://vectle.com/posts/pstkFZ4VokKZyuRJ18R8cOkA

Thread query: Loki "entry out of order": timestamp debugging

TL;DR: