# Loki "entry out of order": timestamp debugging

TL;DR: Loki rejects any log line older than the newest line it already holds for that exact stream. That almost always means two writers are pushing the same stream, or a writer's clock jumped backward. Find and remove the duplicate writer first.

```text
entry out of order
```

## Steps

1. Identify the stream from the rejected batch. The error or the client log names the labels.
**Expected:** You know exactly which stream is rejecting.

2. Hunt for duplicate writers. Two collectors tailing the same file is the top cause: a DaemonSet plus a manually installed agent, overlapping file globs, or a sidecar duplicating the node's collector.
**Expected:** Every file path is tailed by exactly one collector.

3. Check clocks. Compare time on the writers and on Loki; NTP or chrony should be in sync everywhere. A writer whose clock runs behind (or jumped back after a reboot) produces stale timestamps.
**Expected:** Clocks agree within a second or two.

4. If some disorder is unavoidable, enable out-of-order ingestion in the limits config with a small window. The exact key depends on your Loki version, so check its docs for the current name.
**Expected:** Mildly disordered batches are accepted instead of rejected.

5. Check for retry storms. A push that keeps failing and retrying with old batches multiplies the rejects; fix the underlying push failure and the out-of-order noise usually stops with it.
**Expected:** Reject rate drops to near zero.

## Use this when
- Loki push or distributor logs, or client errors, show `entry out of order`

## Not for this skill when
- The error is `too many labels`. That is cardinality, a different limit
- The error is rate limiting on ingestion. That is volume, not ordering
- The error is a stream count limit. That is too many streams, not disorder

## Variant phrasings
### rpc error: code = Unknown desc = entry out of order
The gRPC-wrapped form of the same rejection, seen in distributor logs.

## Why it happens
Loki stores each stream's chunks in timestamp order. An entry older than the stream's newest stored entry cannot be slotted into the already-written position, so it is rejected unless out-of-order ingestion is explicitly enabled.

## Edge cases
- Out-of-order acceptance costs extra memory and CPU; keep the window small
- Duplicate lines from at-least-once delivery are normal and get dropped quietly; a few rejects are not a problem, a flood is
- Restarting an agent can re-send its buffer with old timestamps; that burst of rejects after a restart is expected and harmless

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kFZ4VokKZyuR_J18R8cOkA