kafka consumer "OffsetOutOfRangeException" reset strategy

TL;DR

The consumer's committed offset points at log segments that retention already deleted. Pick a reset policy: earliest replays from the oldest retained offset, latest skips to the end, none throws so you can handle it yourself.

Error

org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: [orders-0]

Steps

1. Set auto.offset.reset in the consumer config. This is the policy Kafka uses when the committed offset is gone:

auto.offset.reset=earliest

Expected output: the consumer restarts from the oldest available offset instead of crashing.

2. Choose deliberately. earliest replays everything retention still holds (safe, can be a huge backlog). latest skips to the end (fast, drops unprocessed messages). none throws OffsetOutOfRangeException so your code can decide per partition.

Expected output: a conscious tradeoff instead of the library default surprising you.

3. For a one-off repair without changing config, seek manually after partition assignment:

consumer.seek(partition, earliest_available_offset)

Expected output: consumption resumes from the offset you chose.

4. Prevent recurrence: alert on consumer lag so a stalled consumer is noticed before retention deletes its offset.

Expected output: the outage that caused this gets caught in hours, not after the retention window passes.

When to use

• A consumer restarts after a long downtime and immediately throws OffsetOutOfRangeException
• Retention was shortened and consumers fell behind
• You are deciding the reset policy for a new consumer group

When not to use

• Rebalance loops or stuck partitions (thats group coordination, not offsets)
• Deserialization errors on current offsets (the offset exists; the bytes are the problem)
• You need exactly-once guarantees without idempotent handling (replay breaks that; fix the handling first)

Variant phrasings

"no offset stored" for a brand-new consumer group

Same setting applies: with no committed offset at all, auto.offset.reset decides where a new group starts.

out of range on a compacted topic

Compaction also removes offsets. earliest replays from the oldest compacted offset, which is usually what you want.

Why it happens

Kafka deletes log segments past the retention window, and committed offsets are just pointers into those segments. When the pointer's segment is gone, the offset is meaningless and the broker refuses it. The reset policy tells the consumer where to go instead.

Edge cases

• earliest after a month-long outage can replay millions of messages; make sure downstream handling is idempotent first.
• none is the right choice for financial or exactly-once pipelines where silent skipping or replaying is unacceptable, but you must catch the exception.
• Changing the policy does not move already-committed valid offsets; it only applies when the stored offset is missing or out of range.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstE2aa9zf8V-1JPmZnEy0iA