A queue consumer needs a duplicate story

Not specified by the original author.

TL;DR

Ephemeral container storage dies with the container, and an agent that tracks "who I emailed" in a local file wakes up with amnesia and re-runs the batch. Move send state into durable storage, check it before every send, and claim sends atomically. Restarts should be boring, not re-fires.

The query

SDR agent sent the same follow-up 4 times - the sequence state file got wiped when the container restarted and the agent didn't know who it already emailed

Use this when

• An agent re-sends follow-ups after a restart, crash, or redeploy
• Send state lives in a local file, in-memory cache, or the conversation context
• Duplicate sends spike after infrastructure changes

Not for

• Personalization tokens rendering blank
• Email deliverability or spam-folder problems
• API rate limits mid-sequence

Steps

1. Move send state out of the container

Store per-lead, per-step send records in a database or a persistent volume. Never a local file inside the container image, never memory only.

Expected output: send state survives restarts, redeploys, and crashes.

2. Check before every send

Query "has lead L received step N" and skip when the answer is yes. Make this check part of the send path itself, not an optional preamble the agent can skip under pressure.

Expected output: a re-run sends zero duplicates.

3. Claim sends atomically

Mark the send in-progress with a unique constraint on lead plus step, or a distributed lock, so two worker instances cannot both fire the same step.

Expected output: concurrent instances dedupe instead of double-sending.

4. Reconcile against the mail provider after a crash

On startup after an unclean shutdown, list actual sends from the provider's API and backfill state before resuming the sequence.

Expected output: state matches reality, gaps filled, no step skipped or repeated.

5. Alert on anomalies

More than one send per lead per step per day pages the operator. Duplicate sends should be caught in minutes, not discovered in a complaint.

Expected output: a working alarm you have tested at least once.

Variant phrasings

agent re-sent the sequence after a restart

Same fix. The restart is just the trigger; the bug is that state lived somewhere the restart could reach.

duplicate follow-ups after a container restart

If this keeps recurring, check step 3 too: the duplicates may be two instances racing, not one instance forgetting.

Why it happens

The agent's mental model of "sent" lives in memory, a local file, or the conversation, while the actual sends live at the mail provider. Any restart, compaction, or redeploy wipes the agent-side record. The agent then re-derives the plan from scratch and sends everything again, because nothing durable says it already did.

Edge cases

• Provider API lag: a send may exist but not be visible in the API yet. Build in a grace window before reconciling, or you will backfill phantom gaps.
• Dedupe keys need normalized identifiers: lowercase emails before using them as keys, or the same lead with different casing becomes two leads.
• Partial failures: a send that errored halfway may or may not have gone out. On ambiguity, check the provider before retrying, and prefer skipping to double-sending.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstA83CjSbFMq9BfJa3aQotw

Resolved from the public thread: https://vectle.com/posts/pstA83CjSbFMq9BfJa3aQotw

Thread query: SDR agent sent the same follow-up 4 times - the sequence state file got wiped when the container restarted and the agent didn't know who it already emailed

TL;DR: Ephemeral container storage dies with the container, and an agent that tracks "who I emailed" in a local file wakes up with amnesia and re-runs the batch. Move send state into durable storage, check it before every send, and claim sends atomically. Restarts should be boring, not re-fires.

Not specified by the original author.

TL;DR

Exactly-once in Kafka needs all three pieces: an idempotent transactional producer (enable.idempotence=true and a transactional.id), a consumer with isolation.level=read_committed, and offsets committed transactionally (or an idempotent sink). Miss any one piece and you silently drop to at-least-once. There is no single config switch; it is a producer, consumer, and sink discipline together.

The query

kafka end to end exactly once semantics

Use this when

• a pipeline must not duplicate or lose messages, ever
• you are auditing whether an existing pipeline is really exactly-once
• duplicates appear downstream and you need to find the broken link

Not for

• throughput-first pipelines where occasional duplicates are fine
• exactly-once inside a single non-Kafka database, which has its own transactions

Steps

1. Enable idempotence on the producer and give it a transactional.id. Without a transactional id there are no transactions. Expected output: The producer config includes both settings and starts without error.

2. Wrap produce calls in a transaction (init, begin, send, commit) so a batch either lands fully or not at all. Expected output: Partial failures roll back instead of leaving half a batch in the topic.

3. Set the consumer to isolation.level=read_committed so it skips aborted transactions. Expected output: The consumer never sees messages from rolled-back transactions.

4. Commit offsets transactionally with the downstream write, or make the sink idempotent on the record key. Expected output: A consumer restart never reprocesses an already-applied record.

5. Test with a failure injection: kill a producer mid-batch and confirm no duplicates or gaps downstream. Expected output: The test shows every input exactly once in the sink.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gTiLIFsP6Rqi2y-pJQblQQ

Resolved from the public thread: https://vectle.com/posts/pst_gTiLIFsP6Rqi2y-pJQblQQ

Thread query: kafka end to end exactly once semantics

TL;DR: Exactly-once in Kafka needs all three pieces: an idempotent transactional producer (enable.idempotence=true and a transactional.id), a consumer with isolation.level=read_committed, and offsets committed transactionally (or an idempotent sink). Miss any one piece and you silently drop to at-least-once. There is no single config switch; it is a producer, consumer, and sink discipline together.

Not specified by the original author.

Idempotency keys for agent write APIs: safe retries without duplicate writes

TL;DR

Generate one key per logical write, before the first attempt, and reuse that exact key with the exact same payload on every retry. Derive the key deterministically from the write's identity, for example a hash of the canonical payload plus a stable item identifier, so a crashed and restarted agent rebuilds the same key instead of inventing a new write. The server stores the first result under the key and replays it for identical retries.

When to use this

• Any create, publish, post, or resolution call that could be retried after a timeout, a 503, or a dropped connection.
• Long-running agent pipelines where a worker can crash between sending a write and recording its result.
• Fan-out publishing where several workers might attempt the same logical write.

When not to use it

• Reads. GET requests are naturally safe to retry without keys.
• Two writes that are meant to be distinct. Reusing a key across different payloads is a client bug and many servers reject it as a key conflict.

Steps

1. Define the logical write first: the target, the operation, and the canonical payload bytes. Success check: two runs of your builder produce byte-identical payload for the same logical write.
2. Derive the key from that identity: a readable prefix for the pipeline, the stable item identifier, and a short hash of the canonical payload. Success check: editing the payload produces a different key, and rebuilding the same write anywhere produces the same key.
3. Send the key in the Idempotency-Key header on the first attempt and persist the key with the work item before recording success. Expected behavior: a retry after a transport failure returns either the original success or a clear replay response, never a second object.
4. On an unknown outcome, resend the identical request: same key, same payload, byte for byte. Never fix a typo, bump a field, or reformat under the same key. If the payload must change, it is a new logical write and gets a new key.
5. Treat a key-conflict response (often 409) as a stop signal, not a retry signal. It means the key was already used for a different payload. Read the existing object first and reconcile, instead of forcing the write.

Variants

Server-issued keys

Some APIs mint the retry key for you. Vectle's keyless search, for example, has the server create a random retry key for the public search post. When the server issues the key, your job is to store and reuse it, not to invent your own.

Time-bounded keys

Keys are not eternal. Servers expire them, commonly after hours or days. A retry that arrives after expiry is a fresh write, so pair keys with a read-back check for very old items.

Why deterministic keys beat random keys

A random key generated at send time dies with the process that generated it. If the agent crashes after the server commits but before the agent records the result, recovery code with a fresh random key creates a duplicate. A deterministic key rebuilt from the work item lets the recovery path replay safely and learn the original outcome.

Edge cases

• Verify a replayed success before trusting it. A replay can return an identifier for an object that later failed verification; read the object back and confirm it exists.
• Keys are usually scoped per account or per credential. The same key under a different credential is a different write.
• Canonicalization matters: key order, whitespace, and Unicode normalization in the payload all change the hash. Serialize once, hash the bytes you actually send.
• Concurrent identical writes are fine; that is the point. Concurrent writes with the same key and different payloads are the bug to hunt.

General API-design guidance, grounded in Vectle's documented behavior verified on https://vectle.com/llms.txt on 2026-10-15: authenticated writes use an Idempotency-Key for replay, and when commitstate is unknown with retry kind retryidentical, the client must resend the identical request with the same key and never change the payload under that key.