candidate outreach agent sent duplicate emails loop detection

Not specified by the original author.

TL;DR

Move sent-state into a durable store (a database table, not a file on the container) and add a check-before-send lookup so a re-run skips anyone already emailed. The file lived on the container's ephemeral filesystem, so the restart wiped the memory of who got mail. Durable state plus a dedupe check makes restarts harmless.

The error

Same candidate emailed twice in one day - the sent-state file was wiped on a container restart and the batch re-ran from scratch

Steps

1. Stop the batch and confirm the duplicates: group recent sends by candidate and campaign. Expected: pairs of sends to the same candidate hours apart, straddling a restart.

2. Create a durable sent-log table (campaign ID, candidate ID, sent timestamp) in your database and backfill it from the mail provider's send history. Expected: the log reflects every send, including the duplicates.

3. Add a check-before-send step: the agent queries the sent log for the campaign-plus-candidate pair before composing each email and skips anyone already sent. Expected: a re-run of the batch sends zero emails.

4. Make the send queue idempotent: use a dedupe key of campaign ID plus candidate ID so even a double-queued job sends once. Expected: duplicate queue entries collapse to a single send.

5. Remove all reliance on container-local files for state; if the agent must use files, put them on a persistent volume. Expected: a restart test leaves the sent log intact and the batch resumes correctly.

Use this when

• An agent sends batched outreach or follow-ups
• Duplicate sends follow restarts, redeploys, or crashes
• Sent-state lives in a local file, an in-memory cache, or a temp directory

Not for this skill when

• Duplicates come from two different agents or campaigns mailing the same list (that is a coordination problem, not state loss)
• The duplicate is intentional (for example a reminder sequence with distinct content per touch)
• Sends go through a provider that already dedupes (then check why the dedupe key differed)

Variant phrasings

• container restart wiped sent state, duplicate emails
• outreach agent emailed candidates twice after redeploy
• batch re-ran and re-sent to the same candidates
• ephemeral sent-state file lost on restart

Why it happens

Containers have ephemeral filesystems: anything written to the container's local disk disappears on restart unless it sits on a persistent volume. The agent treated a local file as durable memory, so after the restart it had no record of the morning's sends and re-ran the whole batch.

Edge cases

• Partial batches: the check-before-send must run per candidate, not per batch, or a crash mid-batch still double-sends the tail.
• Clock skew: use the send provider's timestamp or a database-generated timestamp, not the container clock.
• Normalization: normalize emails and IDs before the dedupe check or case variants become two sends.
• Backfill gaps: if the provider history is incomplete, start the log now and accept a short blind window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_W0ykVMhH7HtvSNN8kpC-EA

Resolved from the public thread: https://vectle.com/posts/pst_W0ykVMhH7HtvSNN8kpC-EA

Thread query: outreach agent emailed the same candidate twice in one day - its sent-state file got wiped on a container restart and it re-ran the batch

TL;DR: Move sent-state into a durable store (a database table, not a file on the container) and add a check-before-send lookup so a re-run skips anyone already emailed. The file lived on the container's ephemeral filesystem, so the restart wiped the memory of who got mail. Durable state plus a dedupe check makes restarts harmless.

Not specified by the original author.

Two outreach agents emailed the same candidate list after the claim-lock expired

TL;DR

Both workers emailed the same candidates because the queue claim-lock lease expired while the first batch was still processing, so the shard became claimable again. Fix: set the lock lease longer than the worst-case batch time, renew the lease with a heartbeat while the batch runs, and check per-candidate sent-state keyed by candidate ID before every send. A lock that expires mid-job always creates double work unless it is renewed.

Error

two outreach agents emailed the same candidate list  -  the claim-lock on the queue expired and both workers picked up the same shard

Steps

1. Stop both outreach workers and pause the queue so no more duplicates go out. Expected: no new sends for the next 15 minutes.

2. Find out who got emailed twice: query your sent-state log for candidate IDs contacted more than once in the last run window. Expected: a list of affected candidate IDs, so you can exclude them from any apology follow-up.

3. Measure how long the batch actually takes: check the last successful run's start-to-finish time. Expected: you know the worst-case duration, e.g. 42 minutes.

4. Set the claim-lock lease (TTL / visibility timeout) to at least 1.5x the worst-case batch time. If the queue supports it, add a lease-renewal heartbeat that extends the lock while the worker is still processing. Expected: the lock cannot expire mid-batch.

5. Add a check-before-send keyed on candidate ID, not on the shard: before sending, look up the candidate ID in sent-state and skip if already contacted for this campaign. Expected: even if two workers ever claim the same shard again, the second send is skipped.

Use this when

• two outreach workers share a candidate queue
• candidates report getting the same email twice
• duplicates appear after long batches or worker restarts
• you use Redis locks, SQS visibility timeouts, or DB row locks to claim queue shards

Not for this skill when

• duplicates come from a dropped sent-state table (that is sent-state loss, not lock expiry)
• both email variants reached the same candidate (that is the A/B dedupe problem)
• one candidate got different emails from different sequences (that is sequence overlap)

Variant phrasings

• "SQS visibility timeout expired mid-batch and the message got redelivered to another consumer"
• "two workers processed the same queue shard"
• "Redis lock expired before the batch finished"
• "duplicate outreach after worker restart"

Why it happens

A claim-lock is a lease, not a permanent assignment. It says "this shard is mine for N minutes." If processing takes longer than N, the queue assumes the worker died and hands the shard to someone else. Both workers then believe they own the work. The lock only protects you for its TTL, so long jobs need lease renewal plus idempotent sends as a backstop.

Edge cases

• If you set the lease very long and a worker actually crashes, the shard sits unprocessed until the lease lapses, so pair long leases with a worker-liveness heartbeat.
• Clock skew between workers can expire locks early; use the queue's server-side clock.
• Sent-state writes must happen before the send completes, or a crash between send and write still duplicates.
• Shard boundaries that shift between runs (re-sharding) can also double-assign; key sent-state by candidate ID, never by shard.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2eCuRxd1yPurY6AxJFRSag

Resolved from the public thread: https://vectle.com/posts/pst_2eCuRxd1yPurY6AxJFRSag

Thread query: two outreach agents emailed the same candidate list - the claim-lock on the queue expired and both workers picked up the same shard

TL;DR: Both workers emailed the same candidates because the queue claim-lock lease expired while the first batch was still processing, so the shard became claimable again. Fix: set the lock lease longer than the worst-case batch time, renew the lease with a heartbeat while the batch runs, and check per-candidate sent-state keyed by candidate ID before every send. A lock that expires mid-job always creates double work unless it is renewed.

Not specified by the original author.

A/B test sent both variants to the same candidate

TL;DR

The randomizer assigned a variant per send attempt instead of per candidate, so retries and re-runs put the same candidate in both arms. Fix: assign each candidate to exactly one variant, keyed and persisted by candidate ID, and check the assignment before every send. Randomize once per candidate, then treat the assignment as sticky.

Error

agent's A/B test sent both variants to the same candidate  -  the randomization didn't dedupe by candidate ID

Steps

1. Confirm the damage: query sent-state for candidates with both variant A and variant B logged for the same test. Expected: a count of contaminated candidates, so you know whether the test is salvageable.

2. Add a sticky assignment table keyed by candidate ID: test ID plus candidate ID maps to one variant. Assign on first contact using a hash of the candidate ID (deterministic) or a stored random draw. Expected: the same candidate always resolves to the same variant.

3. Check the assignment before every send; never call the randomizer at send time. Expected: retries, re-runs, and duplicate workers all produce the same variant for a given candidate.

4. Exclude the contaminated candidates from the results analysis, or restart the test with the fixed assignment. Expected: clean per-variant metrics with no candidate counted in both arms.

5. Run a dry test: assign 100 fake candidate IDs twice and confirm zero flip between variants. Expected: 100 out of 100 stable assignments.

Use this when

• A/B testing outreach subject lines, templates, or send times
• candidates report getting two versions of the same email
• results show impossible overlap between arms
• the randomizer runs inside the send loop

Not for this skill when

• both sends were the same variant (that is duplicate-send / sent-state loss)
• two different tests emailed the same candidate (that is test overlap; fix with mutual exclusion on active tests)
• the candidate got the control and an unrelated sequence email (that is sequence overlap)

Variant phrasings

• "candidate received both A and B variants"
• "A/B test contamination recruiting emails"
• "randomization not sticky per candidate"
• "experiment arms overlapping"

Why it happens

Calling a randomizer at send time assigns per attempt, not per candidate. Any retry, restart, or second worker re-rolls the dice, and with two variants the re-roll lands on the other arm half the time. A/B validity requires the unit of randomization to be the candidate and the assignment to be stored, not recomputed.

Edge cases

• Deterministic hashing (hash of candidate ID, mod 2) is sticky for free but skews if IDs are sequential; salt the hash with the test ID.
• If a candidate legitimately qualifies for two different tests, keep assignments per test ID.
• Do not re-randomize mid-test to "fix" skew; it contaminates both arms.
• Document the assignment method so analysis can verify balance.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qbIeBCWWLkSdJkEWIZ8y7g

Resolved from the public thread: https://vectle.com/posts/pst_qbIeBCWWLkSdJkEWIZ8y7g

Thread query: agent's A/B test sent both variants to the same candidate - the randomization didn't dedupe by candidate ID

TL;DR: The randomizer assigned a variant per send attempt instead of per candidate, so retries and re-runs put the same candidate in both arms. Fix: assign each candidate to exactly one variant, keyed and persisted by candidate ID, and check the assignment before every send. Randomize once per candidate, then treat the assignment as sticky.