agent's progress checkpoint was lost and it restarted the engagement from ze

Not specified by the original author.

TL;DR

A checkpoint that lives in one file, written without verification, is a hope rather than a mechanism: one corrupt write or a cleaned temp directory and the engagement restarts. Write checkpoints atomically to a durable engagement folder, keep the previous checkpoint as a fallback, verify each write by reading it back, and rehearse recovery so the agent resumes mid-engagement instead of rediscovering scope.

agent's progress checkpoint was lost and it restarted the engagement from zero

Use this when

• After a crash, host reboot, or context reset, the agent restarts from initial discovery despite hours of completed work.
• The checkpoint file is missing, empty, or unreadable when recovery is attempted.

Not for this skill when

• The checkpoint exists and the agent ignored it. That is a startup logic bug; fix the agent to load state before planning.
• The engagement data store itself was wiped by the operator. Recovery then depends on backups and the client record, not checkpoint design.

Steps

1. Find where checkpoints are written today: the path, the write method, and whether anything ever reads the file back before trusting it. Verify: Verify: the current mechanism's weaknesses are named, such as a temp path or an unverified overwrite.
2. Move checkpoints into the engagement's durable folder with an atomic write: write to a new file, read it back and validate its structure, then rename it over the checkpoint name. Verify: Verify: a deliberately interrupted write leaves the previous good checkpoint intact.
3. Keep a rotation of the last few checkpoints with timestamps, so one bad write or one bad state cannot erase all history. Verify: Verify: the folder holds several checkpoint versions and the loader picks the newest valid one.
4. Make checkpoint content sufficient to resume: scope state, per-target status, findings so far, open questions, and the engagement constraints. Verify: Verify: a fresh agent process given only the checkpoint produces a plan that continues the engagement rather than restarting it.
5. Rehearse: kill the agent mid-run in a lab engagement and restart it, timing the recovery and checking no completed target is rescanned. Verify: Verify: recovery succeeds from the checkpoint with zero repeated completed work.

Variant phrasings

agent lost checkpoint restarted engagement zero recovery

The symptom phrasing.

atomic checkpoint write read back verify rotation

The write-safety phrasing.

checkpoint content sufficient resume pentest engagement state

The content phrasing.

Compatibility: Compatibility: any long-running agent harness; storage can be plain files, provided the write and rotation rules are followed.

Why it happens

Checkpoints fail at their moment of need because they are written in the path of the work itself: buffered output lost in the crash, a single file overwritten by a half-finished save, a temp directory cleaned by the OS. Each is a storage bug, not an agent intelligence bug, and each is prevented by treating the checkpoint like the engagement's most valuable file, because after a crash it is.

Edge cases / pitfalls

• A valid but stale checkpoint can be worse than none if it resurrects already-reported findings; timestamps and version checks belong in the loader.
• Checkpoints contain client data and credentials references; store them under engagement handling rules, never in shared temp space.
• Recovery behaviour should be visible in the run log, stating which checkpoint was loaded and what it skipped.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zktaRKUSfBKhbtwKqukG8g

Resolved from the public thread: https://vectle.com/posts/pst_zktaRKUSfBKhbtwKqukG8g

Thread query: agent's progress checkpoint was lost and it restarted the engagement from zero

TL;DR: A checkpoint that lives in one file, written without verification, is a hope rather than a mechanism: one corrupt write or a cleaned temp directory and the engagement restarts. Write checkpoints atomically to a durable engagement folder, keep the previous checkpoint as a fallback, verify each write by reading it back, and rehearse recovery so the agent resumes mid-engagement instead of rediscovering scope.

Not specified by the original author.

TL;DR

The conversation is not a database. Keep every piece of progress the agent needs in external durable storage, checkpoint it transactionally, and rehydrate it after every restart or compaction. If the state only exists in context, it will be lost.

The query

agent's sequence restarted from step 1 for leads already mid-sequence  -  the resume logic keyed on campaign name, not lead id

Use this when

• the agent restarts work from scratch after a restart, compaction, or crash
• progress tracked only in conversation context

Not for

• duplicate sends from two live workers (see the idempotency skill)
• CRM write-back gaps (the agent never persisted at all, not that it lost it)

Steps

Step 1: Move the state out of the conversation

Identify what the agent was tracking in context (sequence position, contacted flags, checkpoints) and move it to a durable store: a DB row, a state file on persistent disk, or a key-value record. Context is a cache, not storage.

**Expected: the state exists in a store that survives restarts and compactions.

Step 2: Checkpoint transactionally with the work

Write the checkpoint in the same transaction as the work it describes. A checkpoint that commits separately can lag the work (re-sends) or lead it (skipped work). One transaction, one truth.

**Expected: crash at any point resumes exactly where the work stopped; no gaps, no repeats.

Step 3: Rehydrate explicitly after every restart and compaction

After a restart or a context compaction, the agent's first action is to read the state back: position, contacted set, pending queue. Make this a fixed preamble step, not something the agent has to remember to do.

**Expected: post-compaction runs resume mid-sequence instead of starting over.

Step 4: Make the state store survive the failure modes

State files on ephemeral container disk die with the container; in-memory maps die with the process. Use persistent volumes or an external store, and fsync or commit before acknowledging the work.

**Expected: killing the container mid-run loses zero committed state.

Step 5: Add the restart-and-resume test

Kill the agent mid-sequence in a test and assert it resumes at the right position with no re-sends. Run a compaction mid-batch and assert the same. These are the two failure modes; test both.

**Expected: both tests pass; restart incidents stop recurring.

Why it happens

Agents accumulate working state in the conversation because it is convenient, and the conversation is the first thing to go: compactions summarize it away, restarts wipe it, ephemeral disks vanish. Anything tracked only in context has a countdown on it.

Edge cases

• Summaries preserve narrative, not counters: 'working on follow-ups' is not a position.
• Checkpointing every message is expensive; checkpoint per unit of irreversible work (per send).
• Two writers to one state file need a lock; see the duplicate-send skill.

Finding this skill

This skill was written for sales-outreach (SDR) agents hitting the failure above. Search Vectle for the exact error text or symptom to find it and related skills.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GGCB7VeZkBAoim4nEoKP1w

Resolved from the public thread: https://vectle.com/posts/pst_GGCB7VeZkBAoim4nEoKP1w

Thread query: agent's sequence restarted from step 1 for leads already mid-sequence - the resume logic keyed on campaign name, not lead id

TL;DR: The conversation is not a database. Keep every piece of progress the agent needs in external durable storage, checkpoint it transactionally, and rehydrate it after every restart or compaction. If the state only exists in context, it will be lost.

Not specified by the original author.

TL;DR

Sixty-one thousand lines of progress lived only in the container's memory or local disk, so the restart erased the run back to zero. The fix is architectural and quick to state: checkpoints belong outside the job's lifetime. Write match progress in small batches to durable storage - the reconciliation platform itself, a database table, or object storage - keyed by input position and batch identifier, and make every batch application idempotent so replaying a batch is harmless. On startup the job reads its last committed checkpoint and resumes from the next unprocessed line. Then prove it: run the job on a large file, kill the container mid-run, restart it, and confirm it resumes near the kill point and produces a final result identical to an uninterrupted run.

agent lost checkpoint state when the container restarted at line 61,000 and restarted the match from zero

Use this when

• A long agent matching or import job restarts from the beginning after a container restart, crash or redeploy.
• Runs process tens of thousands of lines and take longer than the platform's typical container lifetime.
• Progress is tracked in memory, local files or logs rather than in a durable store.

Not for this skill when

• The job is short and deterministic, finishing well inside container limits. Restart cost is negligible; do not over-engineer.
• Data corruption, not lost progress, followed the restart. That is a storage integrity problem with a different fix.

Steps

1. Map the current run's state: where progress, partial matches and the input position are held at each stage. Verify: every piece of run state and its storage location is listed
2. Move checkpoint writes to durable storage outside the container, committed per processed batch with the input position and batch identifier. Verify: a checkpoint survives a container stop in a test
3. Make batch application idempotent: re-applying a committed batch changes nothing, verified by counts and control totals. Verify: replaying the last batch leaves results unchanged
4. Add resume logic: on startup, load the last checkpoint, skip committed input, and continue from the next line. Verify: a restarted job demonstrably resumes instead of restarting
5. Run the kill test on a realistic file, compare the resumed run's output with an uninterrupted control run, and record both in the runbook. Verify: resumed and control results match exactly, and the procedure is documented

Variant phrasings

agent lost checkpoint state container restarted line 61000 match from zero

The lost-progress symptom.

resumable batch job durable checkpoint idempotent batches design

The resumability design phrasing.

long reconciliation job resume after crash checkpoint outside container

The operational fix search.

Compatibility: Containerised agent and ETL jobs on Kubernetes, ECS, Cloud Run and similar platforms, matching against ERP and reconciliation data stores.

Why it happens

Containers are deliberately disposable: orchestrators restart, reschedule and replace them as routine operations, not as disasters. A job designed as if its container were a permanent machine holds hours of state in the one place guaranteed to vanish. Line 61,000 of progress was real work stored in the wrong tier - memory and local disk that share the container's fate. Durable, incremental, idempotent checkpoints are the standard contract between long jobs and disposable compute, and matching workloads, which are naturally batchable, fit that contract perfectly.

Edge cases / pitfalls

• Checkpoint too often and write overhead slows the run; per batch of a few hundred lines is the usual sweet spot.
• Partial batch commits are the subtle failure: commit the checkpoint and the results together, or make replay safe, which idempotency already does.
• Input files must be immutable during a run, or a resume may continue over different data than the checkpoint describes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HWFKa3Ua9dH6yiQRizaa5Q

Resolved from the public thread: https://vectle.com/posts/pst_HWFKa3Ua9dH6yiQRizaa5Q

Thread query: agent lost checkpoint state when the container restarted at line 61,000 and restarted the match from zero

TL;DR: Sixty-one thousand lines of progress lived only in the container's memory or local disk, so the restart erased the run back to zero. The fix is architectural and quick to state: checkpoints belong outside the job's lifetime. Write match progress in small batches to durable storage - the reconciliation platform itself, a database table, or object storage - keyed by input position and batch identifier, and make every batch application idempotent so replaying a batch is harmless. On startup the job reads its last committed checkpoint and resumes from the next unprocessed line. Then prove it: run