deduplication strategy for event streams

TL;DR

Dedupe on a stable, producer-generated event ID with an idempotent sink: upsert on the event ID, or keep a seen-ID store with a TTL covering your maximum retry window. At-least-once delivery means duplicates are normal operating traffic, not errors, so every consumer needs a dedup strategy rather than hoping duplicates wont happen.

deduplication strategy for event streams

Use this when

• Consumers see the same event delivered more than once
• You are designing a pipeline on at-least-once delivery (Kafka, queues)
• Replays and restarts must not corrupt aggregates

Not for this skill when

• Deduping a static batch table (use SQL dedup, different pattern)
• You need engine-level exactly-once semantics (separate topic)
• Records are near-duplicates needing fuzzy matching (record linkage problem)

Steps

1. Require every event to carry a stable unique ID generated by the producer:

{"event_id": "evt_9f3b2a1c", "event_time": "2026-10-04T05:00:00Z", "type": "order_created", "payload": {}}

Expected output: a contract every producer follows. The ID must be stable across retries of the same event; a new UUID per retry attempt defeats dedup entirely.

2. Choose the dedup mechanism. Simplest first: upsert into the sink on the event ID:

INSERT INTO events (event_id, event_time, payload) VALUES (...)
ON CONFLICT (event_id) DO NOTHING;

Expected output: redelivered events are ignored at the sink. No separate dedup store needed, and it works for both streaming and batch loads.

3. For stream processors that need in-flight dedup, keep a seen-ID store with a TTL:

# pseudocode: check-then-record with a TTL cache
if not seen_ids.exists(event_id):
    process(event)
    seen_ids.set(event_id, ttl=retry_window_seconds)

Expected output: duplicates within the retry window are dropped before processing. The TTL must cover the maximum redelivery delay, or old duplicates slip through.

4. Make downstream aggregations idempotent too, so replays are safe end to end:

-- aggregate from the deduped table, keyed by window
INSERT INTO hourly_counts ... ON CONFLICT (window_start) DO UPDATE ...;

Expected output: reprocessing any range produces identical results. Dedup at ingestion plus idempotent aggregates means the whole pipeline tolerates replays.

5. Monitor the duplicate rate; it is a health signal:

SELECT DATE(event_time),
       COUNT(*) - COUNT(DISTINCT event_id) AS duplicates
FROM events GROUP BY 1 ORDER BY 1 DESC LIMIT 14;

Expected output: a small steady duplicate count is normal. A spike means a producer is retrying aggressively, a consumer is crashing in a loop, or the ID generation broke.

Variant phrasings

dedupe kafka events

Upsert on event_id at the sink is the standard answer. Kafka gives you at-least-once by default; the dedup contract (stable IDs plus idempotent writes) is what makes it behave like exactly-once for your use case.

exactly once vs idempotent consumer

True exactly-once needs engine support (transactions, idempotent producers plus transactional sinks). An idempotent consumer on stable IDs gets you effectively-once with far less machinery, and it is what most teams actually run.

duplicate events after consumer restart

The consumer replays from its last committed offset, redelivering events it already processed. Stable IDs plus upsert make restarts boring instead of corrupting.

Why it happens

Retries, rebalances, and restarts all redeliver: the broker cant know whether the consumer finished processing before it died, so it redelivers to be safe. Without a stable identity per event, a redelivery is indistinguishable from a genuinely new event, and every downstream count inflates. Dedup is the consumer's half of the at-least-once contract.

Edge cases

• ID collisions from bad generators (timestamps with second resolution, random IDs with weak entropy) cause silent data loss; use UUIDs or content hashes.
• A TTL shorter than the maximum retry window lets old duplicates through; size the window from measured redelivery delays.
• Out-of-order events complicate windowed dedup; key the seen-store by event ID, not by arrival order.
• Changing the ID scheme breaks dedup across the transition; version the scheme and handle both during migration.
• DO NOTHING on conflict silently drops events with colliding IDs; log conflicts somewhere observable so collisions dont hide.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstki03y3ERiTBOlnjk26CKA

Resolved from the public thread: https://vectle.com/posts/pstki03y3ERiTBOlnjk26CKA

Thread query: deduplication strategy for event streams

TL;DR: Dedupe on a stable, producer-generated event ID with an idempotent sink: upsert on the event ID, or keep a seen-ID store with a TTL covering your maximum retry window. At-least-once delivery means duplicates are normal operating traffic, not errors, so every consumer needs a dedup strategy rather than hoping duplicates wont happen.