kafka consumer offset reset strategies

TL;DR

auto.offset.reset only applies when a group has no committed offset: earliest replays from the start, latest starts at the end, none throws an exception instead. For existing groups you reset offsets deliberately with the consumer-groups tooling, ideally with a dry run first. The most common mistake is expecting auto.offset.reset to move an existing group; it never does.

The query

kafka consumer offset reset strategies

Use this when

• a new consumer group starts at the wrong position in the topic
• you need to replay a topic from the beginning or from a timestamp
• committed offsets expired and the group restarted unexpectedly

Not for

• producer idempotence or transaction settings
• rebalancing or partition assignment behavior

Steps

1. Check whether the group has committed offsets. If it does, auto.offset.reset will not fire; you must reset explicitly.

Expected output: You know the group's current committed positions.

1. For a full replay, stop the consumers and reset offsets to earliest (or a timestamp) with the consumer-groups reset command, using dry-run first.

Expected output: The dry run shows exactly which partitions and offsets will change.

1. For skipping poison messages, reset to latest or to a specific offset just past the bad record.

Expected output: Consumers resume after the problem record without replaying everything.

1. Set auto.offset.reset=none in production configs so a missing offset fails loudly instead of silently replaying or skipping.

Expected output: A misconfigured group errors out instead of processing the wrong data.

1. Document every manual reset: who, when, which group, and to what position. Resets are the number one cause of mystery replays.

Expected output: The reset is logged where the team can find it later.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xz-WIi8UclNuwoyYSIZVdg