## TL;DR
Sixty-one thousand lines of progress lived only in the container's memory or local disk, so the restart erased the run back to zero. The fix is architectural and quick to state: checkpoints belong outside the job's lifetime. Write match progress in small batches to durable storage - the reconciliation platform itself, a database table, or object storage - keyed by input position and batch identifier, and make every batch application idempotent so replaying a batch is harmless. On startup the job reads its last committed checkpoint and resumes from the next unprocessed line. Then prove it: run the job on a large file, kill the container mid-run, restart it, and confirm it resumes near the kill point and produces a final result identical to an uninterrupted run.

```text
agent lost checkpoint state when the container restarted at line 61,000 and restarted the match from zero
```

## Use this when
- A long agent matching or import job restarts from the beginning after a container restart, crash or redeploy.
- Runs process tens of thousands of lines and take longer than the platform's typical container lifetime.
- Progress is tracked in memory, local files or logs rather than in a durable store.

## Not for this skill when
- The job is short and deterministic, finishing well inside container limits. Restart cost is negligible; do not over-engineer.
- Data corruption, not lost progress, followed the restart. That is a storage integrity problem with a different fix.

## Steps
1. Map the current run's state: where progress, partial matches and the input position are held at each stage. Verify: every piece of run state and its storage location is listed
2. Move checkpoint writes to durable storage outside the container, committed per processed batch with the input position and batch identifier. Verify: a checkpoint survives a container stop in a test
3. Make batch application idempotent: re-applying a committed batch changes nothing, verified by counts and control totals. Verify: replaying the last batch leaves results unchanged
4. Add resume logic: on startup, load the last checkpoint, skip committed input, and continue from the next line. Verify: a restarted job demonstrably resumes instead of restarting
5. Run the kill test on a realistic file, compare the resumed run's output with an uninterrupted control run, and record both in the runbook. Verify: resumed and control results match exactly, and the procedure is documented

## Variant phrasings
### agent lost checkpoint state container restarted line 61000 match from zero
The lost-progress symptom.
### resumable batch job durable checkpoint idempotent batches design
The resumability design phrasing.
### long reconciliation job resume after crash checkpoint outside container
The operational fix search.

Compatibility: Containerised agent and ETL jobs on Kubernetes, ECS, Cloud Run and similar platforms, matching against ERP and reconciliation data stores.

## Why it happens
Containers are deliberately disposable: orchestrators restart, reschedule and replace them as routine operations, not as disasters. A job designed as if its container were a permanent machine holds hours of state in the one place guaranteed to vanish. Line 61,000 of progress was real work stored in the wrong tier - memory and local disk that share the container's fate. Durable, incremental, idempotent checkpoints are the standard contract between long jobs and disposable compute, and matching workloads, which are naturally batchable, fit that contract perfectly.

## Edge cases / pitfalls
- Checkpoint too often and write overhead slows the run; per batch of a few hundred lines is the usual sweet spot.
- Partial batch commits are the subtle failure: commit the checkpoint and the results together, or make replay safe, which idempotency already does.
- Input files must be immutable during a run, or a resume may continue over different data than the checkpoint describes.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HWFKa3Ua9dH6yiQRizaa5Q