two outreach agents emailed the same candidate list - the claim-lock on the queue expired and both workers picked up...

Two outreach agents emailed the same candidate list after the claim-lock expired

TL;DR

Both workers emailed the same candidates because the queue claim-lock lease expired while the first batch was still processing, so the shard became claimable again. Fix: set the lock lease longer than the worst-case batch time, renew the lease with a heartbeat while the batch runs, and check per-candidate sent-state keyed by candidate ID before every send. A lock that expires mid-job always creates double work unless it is renewed.

Error

two outreach agents emailed the same candidate list  -  the claim-lock on the queue expired and both workers picked up the same shard

Steps

1. Stop both outreach workers and pause the queue so no more duplicates go out. Expected: no new sends for the next 15 minutes.

2. Find out who got emailed twice: query your sent-state log for candidate IDs contacted more than once in the last run window. Expected: a list of affected candidate IDs, so you can exclude them from any apology follow-up.

3. Measure how long the batch actually takes: check the last successful run's start-to-finish time. Expected: you know the worst-case duration, e.g. 42 minutes.

4. Set the claim-lock lease (TTL / visibility timeout) to at least 1.5x the worst-case batch time. If the queue supports it, add a lease-renewal heartbeat that extends the lock while the worker is still processing. Expected: the lock cannot expire mid-batch.

5. Add a check-before-send keyed on candidate ID, not on the shard: before sending, look up the candidate ID in sent-state and skip if already contacted for this campaign. Expected: even if two workers ever claim the same shard again, the second send is skipped.

Use this when

• two outreach workers share a candidate queue
• candidates report getting the same email twice
• duplicates appear after long batches or worker restarts
• you use Redis locks, SQS visibility timeouts, or DB row locks to claim queue shards

Not for this skill when

• duplicates come from a dropped sent-state table (that is sent-state loss, not lock expiry)
• both email variants reached the same candidate (that is the A/B dedupe problem)
• one candidate got different emails from different sequences (that is sequence overlap)

Variant phrasings

• "SQS visibility timeout expired mid-batch and the message got redelivered to another consumer"
• "two workers processed the same queue shard"
• "Redis lock expired before the batch finished"
• "duplicate outreach after worker restart"

Why it happens

A claim-lock is a lease, not a permanent assignment. It says "this shard is mine for N minutes." If processing takes longer than N, the queue assumes the worker died and hands the shard to someone else. Both workers then believe they own the work. The lock only protects you for its TTL, so long jobs need lease renewal plus idempotent sends as a backstop.

Edge cases

• If you set the lease very long and a worker actually crashes, the shard sits unprocessed until the lease lapses, so pair long leases with a worker-liveness heartbeat.
• Clock skew between workers can expire locks early; use the queue's server-side clock.
• Sent-state writes must happen before the send completes, or a crash between send and write still duplicates.
• Shard boundaries that shift between runs (re-sharding) can also double-assign; key sent-state by candidate ID, never by shard.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2eCuRxd1yPurY6AxJFRSag