kafka OffsetOutOfRangeException consumer reset

TL;DR

The consumer's committed offset no longer exists (retention deleted it, or the offset was never valid), so Kafka throws OffsetOutOfRangeException. Decide the recovery policy explicitly: reset to earliest to reprocess, to latest to skip, then set auto.offset.reset to match so it never crashes again.

kafka OffsetOutOfRangeException consumer reset

Use this when

• A Kafka consumer dies with OffsetOutOfRangeException
• The error appears after downtime longer than the retention period
• An agent restarts a consumer group from old offsets

Not for this skill when

• Consumers rebalance constantly (thats session timeouts)
• Messages fail to deserialize (thats the serde)
• The broker is unreachable (thats network)

Steps

1. Inspect the group's offsets vs the topic's actual range:

kafka-consumer-groups.sh --bootstrap-server BROKER --describe --group my-group

Expected output: the committed offset per partition, and the log end offset. If the committed offset is below the log start offset, retention ate it; that gap is the bug.

1. Reset the offset to a valid position. To reprocess from the oldest retained message:

kafka-consumer-groups.sh --bootstrap-server BROKER --group my-group \
  --reset-offsets --to-earliest --topic my-topic --execute

Expected output: offsets reset confirmation. Use --to-latest instead to skip the gap and resume from new messages.

1. Set the auto-reset policy so future occurrences degrade gracefully instead of crashing:

consumer = KafkaConsumer(auto_offset_reset="earliest")  # or "latest"

Expected output: the consumer picks the configured position automatically next time. The default latest silently skips; earliest reprocesses. Choose deliberately.

1. Prevent recurrence by aligning retention with downtime tolerance:

# check retention vs your max expected downtime
kafka-configs.sh --bootstrap-server BROKER --entity-type topics --entity-name my-topic --describe

Expected output: the retention hours. If retention is 24h and the consumer can be down for 48h, this error is guaranteed eventually.

Variant phrasings

offset out of range right after a redeploy

The new consumer group id has no committed offset and the default reset policy picked a bad spot. Set the policy explicitly.

only one partition throws

That partition's committed offset is stale while others are fine. Reset just that partition with --topic my-topic:2 syntax.

Why it happens

Kafka retains messages for a bounded time. A consumer that stays down longer than retention comes back to an offset that no longer exists, and the broker answers with OffsetOutOfRangeException. Without an explicit auto.offset.reset policy, the client throws instead of choosing. Agents hit this restarting long-dead consumers without checking the offset landscape first.

Edge cases

• Resetting to earliest replays everything retained; make the downstream idempotent or you will double-process.
• Compacted topics behave differently: the "earliest" offset may be far from the first live message.
• Committing offsets asynchronously means the stored offset can lag the processed one; on crash you may replay a few messages even without this error.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstCkmS-VfWkDpFD1J41vg