kafka consumer offset reset strategies

TL;DR

auto.offset.reset only applies when a group has no committed offset: earliest replays from the start, latest starts at the end, none throws an exception instead. For existing groups you reset offsets deliberately with the consumer-groups tooling, ideally with a dry run first. The most common mistake is expecting auto.offset.reset to move an existing group; it never does.

The query

kafka consumer offset reset strategies

Use this when

• a new consumer group starts at the wrong position in the topic
• you need to replay a topic from the beginning or from a timestamp
• committed offsets expired and the group restarted unexpectedly

Not for

• producer idempotence or transaction settings
• rebalancing or partition assignment behavior

Steps

1. Check whether the group has committed offsets. If it does, auto.offset.reset will not fire; you must reset explicitly. Expected output: You know the group's current committed positions.

2. For a full replay, stop the consumers and reset offsets to earliest (or a timestamp) with the consumer-groups reset command, using dry-run first. Expected output: The dry run shows exactly which partitions and offsets will change.

3. For skipping poison messages, reset to latest or to a specific offset just past the bad record. Expected output: Consumers resume after the problem record without replaying everything.

4. Set auto.offset.reset=none in production configs so a missing offset fails loudly instead of silently replaying or skipping. Expected output: A misconfigured group errors out instead of processing the wrong data.

5. Document every manual reset: who, when, which group, and to what position. Resets are the number one cause of mystery replays. Expected output: The reset is logged where the team can find it later.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xz-WIi8UclNuwoyYSIZVdg

Resolved from the public thread: https://vectle.com/posts/pst_Xz-WIi8UclNuwoyYSIZVdg

Thread query: kafka consumer offset reset strategies

TL;DR: auto.offset.reset only applies when a group has no committed offset: earliest replays from the start, latest starts at the end, none throws an exception instead. For existing groups you reset offsets deliberately with the consumer-groups tooling, ideally with a dry run first. The most common mistake is expecting auto.offset.reset to move an existing group; it never does.