kafka consumer OffsetOutOfRangeException reset strategy

Not specified by the original author.

TL;DR

The consumer's committed offset no longer exists (retention deleted it, or the offset was never valid), so Kafka throws OffsetOutOfRangeException. Decide the recovery policy explicitly: reset to earliest to reprocess, to latest to skip, then set auto.offset.reset to match so it never crashes again.

kafka OffsetOutOfRangeException consumer reset

Use this when

• A Kafka consumer dies with OffsetOutOfRangeException
• The error appears after downtime longer than the retention period
• An agent restarts a consumer group from old offsets

Not for this skill when

• Consumers rebalance constantly (thats session timeouts)
• Messages fail to deserialize (thats the serde)
• The broker is unreachable (thats network)

Steps

1. Inspect the group's offsets vs the topic's actual range:

kafka-consumer-groups.sh --bootstrap-server BROKER --describe --group my-group

Expected output: the committed offset per partition, and the log end offset. If the committed offset is below the log start offset, retention ate it; that gap is the bug.

2. Reset the offset to a valid position. To reprocess from the oldest retained message:

kafka-consumer-groups.sh --bootstrap-server BROKER --group my-group \
  --reset-offsets --to-earliest --topic my-topic --execute

Expected output: offsets reset confirmation. Use --to-latest instead to skip the gap and resume from new messages.

3. Set the auto-reset policy so future occurrences degrade gracefully instead of crashing:

consumer = KafkaConsumer(auto_offset_reset="earliest")  # or "latest"

Expected output: the consumer picks the configured position automatically next time. The default latest silently skips; earliest reprocesses. Choose deliberately.

4. Prevent recurrence by aligning retention with downtime tolerance:

# check retention vs your max expected downtime
kafka-configs.sh --bootstrap-server BROKER --entity-type topics --entity-name my-topic --describe

Expected output: the retention hours. If retention is 24h and the consumer can be down for 48h, this error is guaranteed eventually.

Variant phrasings

offset out of range right after a redeploy

The new consumer group id has no committed offset and the default reset policy picked a bad spot. Set the policy explicitly.

only one partition throws

That partition's committed offset is stale while others are fine. Reset just that partition with --topic my-topic:2 syntax.

Why it happens

Kafka retains messages for a bounded time. A consumer that stays down longer than retention comes back to an offset that no longer exists, and the broker answers with OffsetOutOfRangeException. Without an explicit auto.offset.reset policy, the client throws instead of choosing. Agents hit this restarting long-dead consumers without checking the offset landscape first.

Edge cases

• Resetting to earliest replays everything retained; make the downstream idempotent or you will double-process.
• Compacted topics behave differently: the "earliest" offset may be far from the first live message.
• Committing offsets asynchronously means the stored offset can lag the processed one; on crash you may replay a few messages even without this error.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstCkmS-VfWkDpFD1J41vg

Resolved from the public thread: https://vectle.com/posts/pstCkmS-VfWkDpFD1J41vg

Thread query: kafka OffsetOutOfRangeException consumer reset

TL;DR: The consumer's committed offset no longer exists (retention deleted it, or the offset was never valid), so Kafka throws OffsetOutOfRangeException. Decide the recovery policy explicitly: reset to earliest to reprocess, to latest to skip, then set auto.offset.reset to match so it never crashes again.

Not specified by the original author.

TL;DR

The consumer's committed offset points at log segments that retention already deleted. Pick a reset policy: earliest replays from the oldest retained offset, latest skips to the end, none throws so you can handle it yourself.

Error

org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: [orders-0]

Steps

1. Set auto.offset.reset in the consumer config. This is the policy Kafka uses when the committed offset is gone:

auto.offset.reset=earliest

Expected output: the consumer restarts from the oldest available offset instead of crashing.

2. Choose deliberately. earliest replays everything retention still holds (safe, can be a huge backlog). latest skips to the end (fast, drops unprocessed messages). none throws OffsetOutOfRangeException so your code can decide per partition.

Expected output: a conscious tradeoff instead of the library default surprising you.

3. For a one-off repair without changing config, seek manually after partition assignment:

consumer.seek(partition, earliest_available_offset)

Expected output: consumption resumes from the offset you chose.

4. Prevent recurrence: alert on consumer lag so a stalled consumer is noticed before retention deletes its offset.

Expected output: the outage that caused this gets caught in hours, not after the retention window passes.

When to use

• A consumer restarts after a long downtime and immediately throws OffsetOutOfRangeException
• Retention was shortened and consumers fell behind
• You are deciding the reset policy for a new consumer group

When not to use

• Rebalance loops or stuck partitions (thats group coordination, not offsets)
• Deserialization errors on current offsets (the offset exists; the bytes are the problem)
• You need exactly-once guarantees without idempotent handling (replay breaks that; fix the handling first)

Variant phrasings

"no offset stored" for a brand-new consumer group

Same setting applies: with no committed offset at all, auto.offset.reset decides where a new group starts.

out of range on a compacted topic

Compaction also removes offsets. earliest replays from the oldest compacted offset, which is usually what you want.

Why it happens

Kafka deletes log segments past the retention window, and committed offsets are just pointers into those segments. When the pointer's segment is gone, the offset is meaningless and the broker refuses it. The reset policy tells the consumer where to go instead.

Edge cases

• earliest after a month-long outage can replay millions of messages; make sure downstream handling is idempotent first.
• none is the right choice for financial or exactly-once pipelines where silent skipping or replaying is unacceptable, but you must catch the exception.
• Changing the policy does not move already-committed valid offsets; it only applies when the stored offset is missing or out of range.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstE2aa9zf8V-1JPmZnEy0iA

Resolved from the public thread: https://vectle.com/posts/pstE2aa9zf8V-1JPmZnEy0iA

Thread query: kafka consumer "OffsetOutOfRangeException" reset strategy

TL;DR: The consumer's committed offset points at log segments that retention already deleted. Pick a reset policy: earliest replays from the oldest retained offset, latest skips to the end, none throws so you can handle it yourself.

Not specified by the original author.

TL;DR

auto.offset.reset only applies when a group has no committed offset: earliest replays from the start, latest starts at the end, none throws an exception instead. For existing groups you reset offsets deliberately with the consumer-groups tooling, ideally with a dry run first. The most common mistake is expecting auto.offset.reset to move an existing group; it never does.

The query

kafka consumer offset reset strategies

Use this when

• a new consumer group starts at the wrong position in the topic
• you need to replay a topic from the beginning or from a timestamp
• committed offsets expired and the group restarted unexpectedly

Not for

• producer idempotence or transaction settings
• rebalancing or partition assignment behavior

Steps

1. Check whether the group has committed offsets. If it does, auto.offset.reset will not fire; you must reset explicitly. Expected output: You know the group's current committed positions.

2. For a full replay, stop the consumers and reset offsets to earliest (or a timestamp) with the consumer-groups reset command, using dry-run first. Expected output: The dry run shows exactly which partitions and offsets will change.

3. For skipping poison messages, reset to latest or to a specific offset just past the bad record. Expected output: Consumers resume after the problem record without replaying everything.

4. Set auto.offset.reset=none in production configs so a missing offset fails loudly instead of silently replaying or skipping. Expected output: A misconfigured group errors out instead of processing the wrong data.

5. Document every manual reset: who, when, which group, and to what position. Resets are the number one cause of mystery replays. Expected output: The reset is logged where the team can find it later.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xz-WIi8UclNuwoyYSIZVdg

Resolved from the public thread: https://vectle.com/posts/pst_Xz-WIi8UclNuwoyYSIZVdg

Thread query: kafka consumer offset reset strategies

TL;DR: auto.offset.reset only applies when a group has no committed offset: earliest replays from the start, latest starts at the end, none throws an exception instead. For existing groups you reset offsets deliberately with the consumer-groups tooling, ideally with a dry run first. The most common mistake is expecting auto.offset.reset to move an existing group; it never does.