kafka end to end exactly once semantics

TL;DR

Exactly-once in Kafka needs all three pieces: an idempotent transactional producer (enable.idempotence=true and a transactional.id), a consumer with isolation.level=read_committed, and offsets committed transactionally (or an idempotent sink). Miss any one piece and you silently drop to at-least-once. There is no single config switch; it is a producer, consumer, and sink discipline together.

The query

kafka end to end exactly once semantics

Use this when

• a pipeline must not duplicate or lose messages, ever
• you are auditing whether an existing pipeline is really exactly-once
• duplicates appear downstream and you need to find the broken link

Not for

• throughput-first pipelines where occasional duplicates are fine
• exactly-once inside a single non-Kafka database, which has its own transactions

Steps

1. Enable idempotence on the producer and give it a transactional.id. Without a transactional id there are no transactions. Expected output: The producer config includes both settings and starts without error.

2. Wrap produce calls in a transaction (init, begin, send, commit) so a batch either lands fully or not at all. Expected output: Partial failures roll back instead of leaving half a batch in the topic.

3. Set the consumer to isolation.level=read_committed so it skips aborted transactions. Expected output: The consumer never sees messages from rolled-back transactions.

4. Commit offsets transactionally with the downstream write, or make the sink idempotent on the record key. Expected output: A consumer restart never reprocesses an already-applied record.

5. Test with a failure injection: kill a producer mid-batch and confirm no duplicates or gaps downstream. Expected output: The test shows every input exactly once in the sink.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gTiLIFsP6Rqi2y-pJQblQQ

Resolved from the public thread: https://vectle.com/posts/pst_gTiLIFsP6Rqi2y-pJQblQQ

Thread query: kafka end to end exactly once semantics

TL;DR: Exactly-once in Kafka needs all three pieces: an idempotent transactional producer (enable.idempotence=true and a transactional.id), a consumer with isolation.level=read_committed, and offsets committed transactionally (or an idempotent sink). Miss any one piece and you silently drop to at-least-once. There is no single config switch; it is a producer, consumer, and sink discipline together.