What determines Kafka consumer offset?

TL;DR: config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use. It is a bit more complex than you described. The

Fix

1. config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use. Expected: this specific failure stops.

2. If you use a high-level java consumer then imagine following scenarios: You have a consumer in a consumer group. Expected: this specific failure stops.

3. Next time you start this consumer it won't even use that. Expected: this specific failure stops.

Details

It is a bit more complex than you described. The auto.offset.reset group1 that has consumed 5 messages and died. auto.offset.reset You have messages in a topic (like you described) and you start a consumer in a new consumer group group2 . There is no offset stored anywhere and this time the auto.offset.reset config will decide whether to start from the beginning of the topic ( earliest ) or from the end of the topic ( latest ) One more thing that affects what offset value will correspond to earliest and latest configs is log retention policy. Imagine you have a topic with retention configured to 1 hour. You produce 5 messages, and then an hour

When to use

You hit exactly this: What determines Kafka consumer offset? in What.

When not to use

A different error, or the same symptom in a different tool. This page only covers the failure above.

Compatibility

What.

Root cause

config and will continue from the place it died because it will just fetch the stored offset from the offset storage (Kafka or ZK as I mentioned).

Asked on Stack Overflow so:32390265: Stack Overflow #32390265 (score 286, accepted answer, 428 votes): I am relatively new to Kafka. I have done a bit of experimenting with it, but a few things are unclear to me regarding consumer offset. From what I have understood so far, when a consumer starts, the offset it will start reading from is determined by the configuration setting auto.offset.reset (correct me if I am wrong). Now say for example that there are 10 messages (offsets 0 to 9) in the topic, and a consumer happened to consume 5 of them before it went down (or before I killed the consumer). Then say I restart that consu The verified fix/answer: It is a bit more complex than you described. The auto.offset.reset config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use. If you use a high-level java consumer then imagine following scenarios:

You have a consumer in a consumer group group1 that has consumed 5 messages and died. Next time you start this consumer it won't even use that auto.offset.reset config and will continue from the place it died because it will just fetch the stored offset from the offset storage (Kafka or ZK as I mentioned).

You have messages in a topic (like you described) and you start a consumer in a new consumer group group2. There is no offset stored anywhere and this time the auto.offset.reset config will decide whether to start from the beginning of the topic (earliest) or from the end of the topic (latest)

One more thing that affects what offset value will correspond to earliest and latest configs is log retention policy. Imagine you have a topic with retention configured to 1 hour. You produce 5 messages, and then an hour Source: https://stackoverflow.com/questions/32390265/what-determines-kafka-consumer-offset