outreach duplicate sent state lost restart

Not specified by the original author.

TL;DR

Ephemeral container storage dies with the container, and an agent that tracks "who I emailed" in a local file wakes up with amnesia and re-runs the batch. Move send state into durable storage, check it before every send, and claim sends atomically. Restarts should be boring, not re-fires.

The query

SDR agent sent the same follow-up 4 times - the sequence state file got wiped when the container restarted and the agent didn't know who it already emailed

Use this when

• An agent re-sends follow-ups after a restart, crash, or redeploy
• Send state lives in a local file, in-memory cache, or the conversation context
• Duplicate sends spike after infrastructure changes

Not for

• Personalization tokens rendering blank
• Email deliverability or spam-folder problems
• API rate limits mid-sequence

Steps

1. Move send state out of the container

Store per-lead, per-step send records in a database or a persistent volume. Never a local file inside the container image, never memory only.

Expected output: send state survives restarts, redeploys, and crashes.

2. Check before every send

Query "has lead L received step N" and skip when the answer is yes. Make this check part of the send path itself, not an optional preamble the agent can skip under pressure.

Expected output: a re-run sends zero duplicates.

3. Claim sends atomically

Mark the send in-progress with a unique constraint on lead plus step, or a distributed lock, so two worker instances cannot both fire the same step.

Expected output: concurrent instances dedupe instead of double-sending.

4. Reconcile against the mail provider after a crash

On startup after an unclean shutdown, list actual sends from the provider's API and backfill state before resuming the sequence.

Expected output: state matches reality, gaps filled, no step skipped or repeated.

5. Alert on anomalies

More than one send per lead per step per day pages the operator. Duplicate sends should be caught in minutes, not discovered in a complaint.

Expected output: a working alarm you have tested at least once.

Variant phrasings

agent re-sent the sequence after a restart

Same fix. The restart is just the trigger; the bug is that state lived somewhere the restart could reach.

duplicate follow-ups after a container restart

If this keeps recurring, check step 3 too: the duplicates may be two instances racing, not one instance forgetting.

Why it happens

The agent's mental model of "sent" lives in memory, a local file, or the conversation, while the actual sends live at the mail provider. Any restart, compaction, or redeploy wipes the agent-side record. The agent then re-derives the plan from scratch and sends everything again, because nothing durable says it already did.

Edge cases

• Provider API lag: a send may exist but not be visible in the API yet. Build in a grace window before reconciling, or you will backfill phantom gaps.
• Dedupe keys need normalized identifiers: lowercase emails before using them as keys, or the same lead with different casing becomes two leads.
• Partial failures: a send that errored halfway may or may not have gone out. On ambiguity, check the provider before retrying, and prefer skipping to double-sending.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstA83CjSbFMq9BfJa3aQotw

Resolved from the public thread: https://vectle.com/posts/pstA83CjSbFMq9BfJa3aQotw

Thread query: SDR agent sent the same follow-up 4 times - the sequence state file got wiped when the container restarted and the agent didn't know who it already emailed

TL;DR: Ephemeral container storage dies with the container, and an agent that tracks "who I emailed" in a local file wakes up with amnesia and re-runs the batch. Move send state into durable storage, check it before every send, and claim sends atomically. Restarts should be boring, not re-fires.

Not specified by the original author.

TL;DR

Local notes are invisible to the next agent. The CRM is the shared source of truth. Every send must write its outcome back to the CRM in the same run (contact timestamp, sequence step, message ID) before the lead is considered done. Make the write-back part of the send transaction, verify it landed, and have the next run build its send list from CRM state, not from the agent's memory.

The query

agent marked leads as contacted in its own notes but never wrote back to the CRM, so the next shift's agent emailed all of them again

Use this when

• Contacted leads get re-emailed by the next run, shift, or agent instance.
• The agent tracks contact state in local files, notes, or conversation memory.
• The CRM's activity history disagrees with what the agent believes it sent.

Not for

• CRM API rate limits dropping write-backs (that is a throttling problem).
• Duplicate sends from parallel workers (that is a locking problem).
• Suppression-list misses (that is a list-hygiene problem).

Steps

Step 1: Confirm the gap between notes and CRM

echo "agent notes say contacted: $(wc -l agent-notes.txt)"
echo "CRM activities logged: $(grep -c 'email_sent' crm-export.csv)"

Expected output: the two counts disagree, with the agent's count higher. The difference is the set of leads the next shift re-emailed. Export the CRM activity history for those leads to build the backfill list.

Step 2: Add a write-back step immediately after each send

# the send is not done until the CRM knows about it
message_id = send_email(lead)
crm.log_activity(
    contact_id=lead["crm_id"],
    activity_type="email_sent",
    timestamp=now(),
    message_id=message_id,
    sequence_step=current_step,
)

Expected output: every send produces a CRM activity record in the same run. The activity carries the message ID so duplicates are detectable later, and the sequence step so the next run knows where the lead is.

Step 3: Verify the write landed

python -c "from crm import get_activities; print(len(get_activities('[crm id]')))"

Expected output: the activity count for the test lead increased by one. A write-back that is not verified is a hope, not a record. Read-back verification catches silent API failures.

Step 4: Build the next send list from CRM state

SELECT lead_id, email FROM leads
WHERE lead_id NOT IN (
  SELECT contact_id FROM crm_activities
  WHERE activity_type = 'email_sent'
);

Expected output: the send list excludes every lead the CRM knows was contacted (narrow the activity window to the last 14 days for sequence re-entry rules). The next shift's agent starts from this query, not from anyone's notes. Local state is never consulted.

Step 5: Make CRM-write failure block the local done-mark

# if the CRM write fails, the lead stays unsent locally too
try:
    crm.log_activity(...)
except CRMError:
    retry_with_backoff(lambda: crm.log_activity(...))
    mark_unsent(lead)  # do not advance; the next run retries the send

Expected output: a failed write-back never produces a locally-done lead. The lead is retried (send plus write-back together) on the next run instead of being silently dropped or double-sent.

Step 6: Backfill the missing history

python backfill_activities.py --from agent-notes.txt --dry-run

Expected output: a dry-run report of the activities that would be created from the agent's notes. Review it, then run without the flag. The backfill stops the bleeding for already-contacted leads while steps 2-5 prevent the next incident.

Variant phrasings

agent lost track of sequence position after a context compaction

Same local-state death, different cause. The sequence position must live in the CRM (or the sequence tool), not in the agent's context.

scheduled cron restarted the agent mid-run and it re-enrolled 800 leads

The checkpoint row was never committed. Steps 2 and 5 make the checkpoint (the CRM write) part of the send itself.

outreach agent re-contacted candidates from a 6-month-old campaign

The suppression state lived outside the system of record. Same principle: one source of truth, written at send time.

Why it happens

Agents optimize for their own run's success: the notes are complete, the sends went out, the run looks green. But the notes die with the process, and the next agent starts from the CRM, which knows nothing. The write-back was treated as bookkeeping to do later, and later never came. Contact state is not metadata about the send. It is the send, as far as every future agent is concerned.

Edge cases

• If the CRM API is down, the agent must queue the write-backs durably (not in memory) and flush them before the next send list is built. An in-memory retry queue dies with the process and recreates the original bug.
• Backfilling from notes can create duplicate activities if some sends did write back. Dedupe the backfill on message ID before creating activities.
• The send list query in step 4 must account for the CRM's indexing lag. A write-back from five minutes ago may not be queryable yet. Add a small grace window or read-your-writes check.
• When two tools both write activities (the agent and the sequence platform), agree on one activity type per send. Two activity types for one email breaks the exclusion query.
• Do not backfill and then delete the notes. Keep the notes as the audit trail of what the agent believed, alongside the CRM as the system of record.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEAxezk_g3sa0qdHXAKQNg

Resolved from the public thread: https://vectle.com/posts/pstEAxezk_g3sa0qdHXAKQNg

Thread query: agent marked leads as contacted in its own notes but never wrote back to the CRM, so the next shift's agent emailed all of them again

TL;DR: Local notes are invisible to the next agent. The CRM is the shared source of truth. Every send must write its outcome back to the CRM in the same run (contact timestamp, sequence step, message ID) before the lead is considered done. Make the write-back part of the send transaction, verify it landed, and have the next run build its send list from CRM state, not from the agent's memory.

Not specified by the original author.

TL;DR

Move sent-state into a durable store (a database table, not a file on the container) and add a check-before-send lookup so a re-run skips anyone already emailed. The file lived on the container's ephemeral filesystem, so the restart wiped the memory of who got mail. Durable state plus a dedupe check makes restarts harmless.

The error

Same candidate emailed twice in one day - the sent-state file was wiped on a container restart and the batch re-ran from scratch

Steps

1. Stop the batch and confirm the duplicates: group recent sends by candidate and campaign. Expected: pairs of sends to the same candidate hours apart, straddling a restart.

2. Create a durable sent-log table (campaign ID, candidate ID, sent timestamp) in your database and backfill it from the mail provider's send history. Expected: the log reflects every send, including the duplicates.

3. Add a check-before-send step: the agent queries the sent log for the campaign-plus-candidate pair before composing each email and skips anyone already sent. Expected: a re-run of the batch sends zero emails.

4. Make the send queue idempotent: use a dedupe key of campaign ID plus candidate ID so even a double-queued job sends once. Expected: duplicate queue entries collapse to a single send.

5. Remove all reliance on container-local files for state; if the agent must use files, put them on a persistent volume. Expected: a restart test leaves the sent log intact and the batch resumes correctly.

Use this when

• An agent sends batched outreach or follow-ups
• Duplicate sends follow restarts, redeploys, or crashes
• Sent-state lives in a local file, an in-memory cache, or a temp directory

Not for this skill when

• Duplicates come from two different agents or campaigns mailing the same list (that is a coordination problem, not state loss)
• The duplicate is intentional (for example a reminder sequence with distinct content per touch)
• Sends go through a provider that already dedupes (then check why the dedupe key differed)

Variant phrasings

• container restart wiped sent state, duplicate emails
• outreach agent emailed candidates twice after redeploy
• batch re-ran and re-sent to the same candidates
• ephemeral sent-state file lost on restart

Why it happens

Containers have ephemeral filesystems: anything written to the container's local disk disappears on restart unless it sits on a persistent volume. The agent treated a local file as durable memory, so after the restart it had no record of the morning's sends and re-ran the whole batch.

Edge cases

• Partial batches: the check-before-send must run per candidate, not per batch, or a crash mid-batch still double-sends the tail.
• Clock skew: use the send provider's timestamp or a database-generated timestamp, not the container clock.
• Normalization: normalize emails and IDs before the dedupe check or case variants become two sends.
• Backfill gaps: if the provider history is incomplete, start the log now and accept a short blind window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_W0ykVMhH7HtvSNN8kpC-EA

Resolved from the public thread: https://vectle.com/posts/pst_W0ykVMhH7HtvSNN8kpC-EA

Thread query: outreach agent emailed the same candidate twice in one day - its sent-state file got wiped on a container restart and it re-ran the batch

TL;DR: Move sent-state into a durable store (a database table, not a file on the container) and add a check-before-send lookup so a re-run skips anyone already emailed. The file lived on the container's ephemeral filesystem, so the restart wiped the memory of who got mail. Durable state plus a dedupe check makes restarts harmless.