kafka topic retention vs compaction choice

TL;DR

Retention deletes old messages by age or size; compaction keeps only the latest value per key, forever. Use retention for event streams like logs and metrics where history ages out. Use compaction for changelog topics like user profiles or config where the latest value per key is the truth. Never compact a stream you need to replay for history, because old versions are gone.

The query

kafka topic retention vs compaction choice

Use this when

• you are creating a topic and unsure whether to compact it
• disk usage on a topic grows without bound
• consumers need the current state per key, not the full history

Not for

• consumer offset reset or retention of the offsets topic itself
• message ordering guarantees, which are a partition concern

Steps

1. Classify the topic: an event stream (append, age out) gets retention; a changelog (key updates, latest wins) gets compaction.

Expected output: You can state which category your topic falls into.

1. For retention, set retention.ms and retention.bytes to match how long consumers actually need to replay.

Expected output: Old segments disappear on schedule and disk usage stays bounded.

1. For compaction, set cleanup.policy=compact and make sure every message has a meaningful key; keyless messages compact badly.

Expected output: The topic keeps one current record per key.

1. Consider compact plus a retention time (cleanup.policy=compact,delete) when you want latest-per-key but also want ancient keys to eventually vanish.

Expected output: Inactive keys expire while active keys stay current.

1. Document the choice next to the topic config so the next person does not flip it casually.

Expected output: The topic's purpose and cleanup policy are written down together.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MgsmM6zo34HJElt8XjLaiA