segment track calls queued during outage and agent replayed them all twice

TL;DR

Attach an idempotency key to every track call before it enters the queue, and advance the replay cursor only after a batch is acknowledged. Replays run as a single marked pass so a second trigger finds nothing to do. Recovery sends each event exactly once.

segment track calls queued during outage and agent replayed them all twice

Use this when

• track calls queue up during an outage or rate-limit pause
• a replay after recovery double-sends events
• dashboards show doubled counts after an incident
• the agent retries batches without knowing what already sent

Not for this skill when

• events are malformed or missing properties
• the destination itself is misconfigured
• you need real-time streaming rather than queued delivery

Steps

1. Attach an idempotency key (event id plus a timestamp bucket) to every track call before it enters the queue. Expected: the receiver dedupes replays automatically.
2. Advance the replay cursor only after the batch is acknowledged, and store the cursor atomically with the batch state. Expected: a crash replays from the last acked batch, never from zero.
3. Run replays as a single pass with a replay marker so a second trigger finds nothing to do. Expected: concurrent or repeated triggers do not double-send.
4. Cap replay batch size and print a dry-run count before the real replay. Expected: you see the pending count before anything sends.
5. After replay, reconcile counts against the receiver's deduped totals. Expected: sent count matches stored count exactly.

Variant phrasings

queued events sent twice after an outage

Short version: idempotency keys plus an atomic cursor make replay exactly-once.

analytics replay double-counted events

Short version: mark the replay pass so a second trigger finds nothing to send.

agent retried batches without tracking what sent

Short version: advance the cursor only on acknowledgment, never on attempt.

Compatibility: applies to Segment and any analytics pipeline with a queue in front (Kafka, SQS, local disk queue); idempotency keys are a sender-side pattern, no SDK version pinned.

Why it happens

During the outage the agent kept appending to the queue while a retry loop also fired, and neither knew what the other had sent. On recovery both paths drained the same backlog. Without idempotency keys the receiver could not tell the two copies apart, so every event counted twice.

Edge cases / pitfalls

• Idempotency keys must be stable across retries; generating a fresh key per attempt defeats the dedupe.
• A cursor stored separately from batch state can drift; write them in one atomic step.
• Capping the buffer means dropping the oldest events in a long outage; emit a dropped-events counter so the gap is visible.
• Reconciling against the receiver takes time on large backlogs; sample first, then do the full count.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lNgihYQ-cRMAdyaavz05Zw