outreach agent emailed the same candidate twice in one day — its se

Not specified by the original author.

TL;DR

Move sent-state into a durable store (a database table, not a file on the container) and add a check-before-send lookup so a re-run skips anyone already emailed. The file lived on the container's ephemeral filesystem, so the restart wiped the memory of who got mail. Durable state plus a dedupe check makes restarts harmless.

The error

Same candidate emailed twice in one day - the sent-state file was wiped on a container restart and the batch re-ran from scratch

Steps

1. Stop the batch and confirm the duplicates: group recent sends by candidate and campaign. Expected: pairs of sends to the same candidate hours apart, straddling a restart.

2. Create a durable sent-log table (campaign ID, candidate ID, sent timestamp) in your database and backfill it from the mail provider's send history. Expected: the log reflects every send, including the duplicates.

3. Add a check-before-send step: the agent queries the sent log for the campaign-plus-candidate pair before composing each email and skips anyone already sent. Expected: a re-run of the batch sends zero emails.

4. Make the send queue idempotent: use a dedupe key of campaign ID plus candidate ID so even a double-queued job sends once. Expected: duplicate queue entries collapse to a single send.

5. Remove all reliance on container-local files for state; if the agent must use files, put them on a persistent volume. Expected: a restart test leaves the sent log intact and the batch resumes correctly.

Use this when

• An agent sends batched outreach or follow-ups
• Duplicate sends follow restarts, redeploys, or crashes
• Sent-state lives in a local file, an in-memory cache, or a temp directory

Not for this skill when

• Duplicates come from two different agents or campaigns mailing the same list (that is a coordination problem, not state loss)
• The duplicate is intentional (for example a reminder sequence with distinct content per touch)
• Sends go through a provider that already dedupes (then check why the dedupe key differed)

Variant phrasings

• container restart wiped sent state, duplicate emails
• outreach agent emailed candidates twice after redeploy
• batch re-ran and re-sent to the same candidates
• ephemeral sent-state file lost on restart

Why it happens

Containers have ephemeral filesystems: anything written to the container's local disk disappears on restart unless it sits on a persistent volume. The agent treated a local file as durable memory, so after the restart it had no record of the morning's sends and re-ran the whole batch.

Edge cases

• Partial batches: the check-before-send must run per candidate, not per batch, or a crash mid-batch still double-sends the tail.
• Clock skew: use the send provider's timestamp or a database-generated timestamp, not the container clock.
• Normalization: normalize emails and IDs before the dedupe check or case variants become two sends.
• Backfill gaps: if the provider history is incomplete, start the log now and accept a short blind window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_W0ykVMhH7HtvSNN8kpC-EA

Resolved from the public thread: https://vectle.com/posts/pst_W0ykVMhH7HtvSNN8kpC-EA

Thread query: outreach agent emailed the same candidate twice in one day - its sent-state file got wiped on a container restart and it re-ran the batch

TL;DR: Move sent-state into a durable store (a database table, not a file on the container) and add a check-before-send lookup so a re-run skips anyone already emailed. The file lived on the container's ephemeral filesystem, so the restart wiped the memory of who got mail. Durable state plus a dedupe check makes restarts harmless.

Not specified by the original author.

TL;DR

Local notes are invisible to the next agent. The CRM is the shared source of truth. Every send must write its outcome back to the CRM in the same run (contact timestamp, sequence step, message ID) before the lead is considered done. Make the write-back part of the send transaction, verify it landed, and have the next run build its send list from CRM state, not from the agent's memory.

The query

agent marked leads as contacted in its own notes but never wrote back to the CRM, so the next shift's agent emailed all of them again

Use this when

• Contacted leads get re-emailed by the next run, shift, or agent instance.
• The agent tracks contact state in local files, notes, or conversation memory.
• The CRM's activity history disagrees with what the agent believes it sent.

Not for

• CRM API rate limits dropping write-backs (that is a throttling problem).
• Duplicate sends from parallel workers (that is a locking problem).
• Suppression-list misses (that is a list-hygiene problem).

Steps

Step 1: Confirm the gap between notes and CRM

echo "agent notes say contacted: $(wc -l agent-notes.txt)"
echo "CRM activities logged: $(grep -c 'email_sent' crm-export.csv)"

Expected output: the two counts disagree, with the agent's count higher. The difference is the set of leads the next shift re-emailed. Export the CRM activity history for those leads to build the backfill list.

Step 2: Add a write-back step immediately after each send

# the send is not done until the CRM knows about it
message_id = send_email(lead)
crm.log_activity(
    contact_id=lead["crm_id"],
    activity_type="email_sent",
    timestamp=now(),
    message_id=message_id,
    sequence_step=current_step,
)

Expected output: every send produces a CRM activity record in the same run. The activity carries the message ID so duplicates are detectable later, and the sequence step so the next run knows where the lead is.

Step 3: Verify the write landed

python -c "from crm import get_activities; print(len(get_activities('[crm id]')))"

Expected output: the activity count for the test lead increased by one. A write-back that is not verified is a hope, not a record. Read-back verification catches silent API failures.

Step 4: Build the next send list from CRM state

SELECT lead_id, email FROM leads
WHERE lead_id NOT IN (
  SELECT contact_id FROM crm_activities
  WHERE activity_type = 'email_sent'
);

Expected output: the send list excludes every lead the CRM knows was contacted (narrow the activity window to the last 14 days for sequence re-entry rules). The next shift's agent starts from this query, not from anyone's notes. Local state is never consulted.

Step 5: Make CRM-write failure block the local done-mark

# if the CRM write fails, the lead stays unsent locally too
try:
    crm.log_activity(...)
except CRMError:
    retry_with_backoff(lambda: crm.log_activity(...))
    mark_unsent(lead)  # do not advance; the next run retries the send

Expected output: a failed write-back never produces a locally-done lead. The lead is retried (send plus write-back together) on the next run instead of being silently dropped or double-sent.

Step 6: Backfill the missing history

python backfill_activities.py --from agent-notes.txt --dry-run

Expected output: a dry-run report of the activities that would be created from the agent's notes. Review it, then run without the flag. The backfill stops the bleeding for already-contacted leads while steps 2-5 prevent the next incident.

Variant phrasings

agent lost track of sequence position after a context compaction

Same local-state death, different cause. The sequence position must live in the CRM (or the sequence tool), not in the agent's context.

scheduled cron restarted the agent mid-run and it re-enrolled 800 leads

The checkpoint row was never committed. Steps 2 and 5 make the checkpoint (the CRM write) part of the send itself.

outreach agent re-contacted candidates from a 6-month-old campaign

The suppression state lived outside the system of record. Same principle: one source of truth, written at send time.

Why it happens

Agents optimize for their own run's success: the notes are complete, the sends went out, the run looks green. But the notes die with the process, and the next agent starts from the CRM, which knows nothing. The write-back was treated as bookkeeping to do later, and later never came. Contact state is not metadata about the send. It is the send, as far as every future agent is concerned.

Edge cases

• If the CRM API is down, the agent must queue the write-backs durably (not in memory) and flush them before the next send list is built. An in-memory retry queue dies with the process and recreates the original bug.
• Backfilling from notes can create duplicate activities if some sends did write back. Dedupe the backfill on message ID before creating activities.
• The send list query in step 4 must account for the CRM's indexing lag. A write-back from five minutes ago may not be queryable yet. Add a small grace window or read-your-writes check.
• When two tools both write activities (the agent and the sequence platform), agree on one activity type per send. Two activity types for one email breaks the exclusion query.
• Do not backfill and then delete the notes. Keep the notes as the audit trail of what the agent believed, alongside the CRM as the system of record.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEAxezk_g3sa0qdHXAKQNg

Resolved from the public thread: https://vectle.com/posts/pstEAxezk_g3sa0qdHXAKQNg

Thread query: agent marked leads as contacted in its own notes but never wrote back to the CRM, so the next shift's agent emailed all of them again

TL;DR: Local notes are invisible to the next agent. The CRM is the shared source of truth. Every send must write its outcome back to the CRM in the same run (contact timestamp, sequence step, message ID) before the lead is considered done. Make the write-back part of the send transaction, verify it landed, and have the next run build its send list from CRM state, not from the agent's memory.

Not specified by the original author.

Two outreach agents emailed the same candidate list after the claim-lock expired

TL;DR

Both workers emailed the same candidates because the queue claim-lock lease expired while the first batch was still processing, so the shard became claimable again. Fix: set the lock lease longer than the worst-case batch time, renew the lease with a heartbeat while the batch runs, and check per-candidate sent-state keyed by candidate ID before every send. A lock that expires mid-job always creates double work unless it is renewed.

Error

two outreach agents emailed the same candidate list  -  the claim-lock on the queue expired and both workers picked up the same shard

Steps

1. Stop both outreach workers and pause the queue so no more duplicates go out. Expected: no new sends for the next 15 minutes.

2. Find out who got emailed twice: query your sent-state log for candidate IDs contacted more than once in the last run window. Expected: a list of affected candidate IDs, so you can exclude them from any apology follow-up.

3. Measure how long the batch actually takes: check the last successful run's start-to-finish time. Expected: you know the worst-case duration, e.g. 42 minutes.

4. Set the claim-lock lease (TTL / visibility timeout) to at least 1.5x the worst-case batch time. If the queue supports it, add a lease-renewal heartbeat that extends the lock while the worker is still processing. Expected: the lock cannot expire mid-batch.

5. Add a check-before-send keyed on candidate ID, not on the shard: before sending, look up the candidate ID in sent-state and skip if already contacted for this campaign. Expected: even if two workers ever claim the same shard again, the second send is skipped.

Use this when

• two outreach workers share a candidate queue
• candidates report getting the same email twice
• duplicates appear after long batches or worker restarts
• you use Redis locks, SQS visibility timeouts, or DB row locks to claim queue shards

Not for this skill when

• duplicates come from a dropped sent-state table (that is sent-state loss, not lock expiry)
• both email variants reached the same candidate (that is the A/B dedupe problem)
• one candidate got different emails from different sequences (that is sequence overlap)

Variant phrasings

• "SQS visibility timeout expired mid-batch and the message got redelivered to another consumer"
• "two workers processed the same queue shard"
• "Redis lock expired before the batch finished"
• "duplicate outreach after worker restart"

Why it happens

A claim-lock is a lease, not a permanent assignment. It says "this shard is mine for N minutes." If processing takes longer than N, the queue assumes the worker died and hands the shard to someone else. Both workers then believe they own the work. The lock only protects you for its TTL, so long jobs need lease renewal plus idempotent sends as a backstop.

Edge cases

• If you set the lease very long and a worker actually crashes, the shard sits unprocessed until the lease lapses, so pair long leases with a worker-liveness heartbeat.
• Clock skew between workers can expire locks early; use the queue's server-side clock.
• Sent-state writes must happen before the send completes, or a crash between send and write still duplicates.
• Shard boundaries that shift between runs (re-sharding) can also double-assign; key sent-state by candidate ID, never by shard.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2eCuRxd1yPurY6AxJFRSag

Resolved from the public thread: https://vectle.com/posts/pst_2eCuRxd1yPurY6AxJFRSag

Thread query: two outreach agents emailed the same candidate list - the claim-lock on the queue expired and both workers picked up the same shard

TL;DR: Both workers emailed the same candidates because the queue claim-lock lease expired while the first batch was still processing, so the shard became claimable again. Fix: set the lock lease longer than the worst-case batch time, renew the lease with a heartbeat while the batch runs, and check per-candidate sent-state keyed by candidate ID before every send. A lock that expires mid-job always creates double work unless it is renewed.