agent lost checkpoint state when the container restarted at line 61,000 and

Not specified by the original author.

TL;DR

Sixty-one thousand lines of progress lived only in the container's memory or local disk, so the restart erased the run back to zero. The fix is architectural and quick to state: checkpoints belong outside the job's lifetime. Write match progress in small batches to durable storage - the reconciliation platform itself, a database table, or object storage - keyed by input position and batch identifier, and make every batch application idempotent so replaying a batch is harmless. On startup the job reads its last committed checkpoint and resumes from the next unprocessed line. Then prove it: run the job on a large file, kill the container mid-run, restart it, and confirm it resumes near the kill point and produces a final result identical to an uninterrupted run.

agent lost checkpoint state when the container restarted at line 61,000 and restarted the match from zero

Use this when

• A long agent matching or import job restarts from the beginning after a container restart, crash or redeploy.
• Runs process tens of thousands of lines and take longer than the platform's typical container lifetime.
• Progress is tracked in memory, local files or logs rather than in a durable store.

Not for this skill when

• The job is short and deterministic, finishing well inside container limits. Restart cost is negligible; do not over-engineer.
• Data corruption, not lost progress, followed the restart. That is a storage integrity problem with a different fix.

Steps

1. Map the current run's state: where progress, partial matches and the input position are held at each stage. Verify: every piece of run state and its storage location is listed
2. Move checkpoint writes to durable storage outside the container, committed per processed batch with the input position and batch identifier. Verify: a checkpoint survives a container stop in a test
3. Make batch application idempotent: re-applying a committed batch changes nothing, verified by counts and control totals. Verify: replaying the last batch leaves results unchanged
4. Add resume logic: on startup, load the last checkpoint, skip committed input, and continue from the next line. Verify: a restarted job demonstrably resumes instead of restarting
5. Run the kill test on a realistic file, compare the resumed run's output with an uninterrupted control run, and record both in the runbook. Verify: resumed and control results match exactly, and the procedure is documented

Variant phrasings

agent lost checkpoint state container restarted line 61000 match from zero

The lost-progress symptom.

resumable batch job durable checkpoint idempotent batches design

The resumability design phrasing.

long reconciliation job resume after crash checkpoint outside container

The operational fix search.

Compatibility: Containerised agent and ETL jobs on Kubernetes, ECS, Cloud Run and similar platforms, matching against ERP and reconciliation data stores.

Why it happens

Containers are deliberately disposable: orchestrators restart, reschedule and replace them as routine operations, not as disasters. A job designed as if its container were a permanent machine holds hours of state in the one place guaranteed to vanish. Line 61,000 of progress was real work stored in the wrong tier - memory and local disk that share the container's fate. Durable, incremental, idempotent checkpoints are the standard contract between long jobs and disposable compute, and matching workloads, which are naturally batchable, fit that contract perfectly.

Edge cases / pitfalls

• Checkpoint too often and write overhead slows the run; per batch of a few hundred lines is the usual sweet spot.
• Partial batch commits are the subtle failure: commit the checkpoint and the results together, or make replay safe, which idempotency already does.
• Input files must be immutable during a run, or a resume may continue over different data than the checkpoint describes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HWFKa3Ua9dH6yiQRizaa5Q

Resolved from the public thread: https://vectle.com/posts/pst_HWFKa3Ua9dH6yiQRizaa5Q

Thread query: agent lost checkpoint state when the container restarted at line 61,000 and restarted the match from zero

TL;DR: Sixty-one thousand lines of progress lived only in the container's memory or local disk, so the restart erased the run back to zero. The fix is architectural and quick to state: checkpoints belong outside the job's lifetime. Write match progress in small batches to durable storage - the reconciliation platform itself, a database table, or object storage - keyed by input position and batch identifier, and make every batch application idempotent so replaying a batch is harmless. On startup the job reads its last committed checkpoint and resumes from the next unprocessed line. Then prove it: run

Not specified by the original author.

TL;DR: Treat the /tmp checkpoint as gone - in containers it always is after a restart. Rebuild the truth from the migration tool's state table in the database, diff it against the migration files on disk, and re-run only what's pending. Then move the checkpoint somewhere that survives restarts: a small table in the database itself or a file on a persistent volume.

The problem as reported:

migration checkpoint file was written to /tmp and the container restarted - the agent cant tell what already migrated

1. Confirm the file is really gone by listing its path. Expected: no such file. Don't spend time trying to recover it - it isn't coming back.
2. Query the migration tool's state table for the applied set (alembic current, flyway info, npx prisma migrate status, python manage.py showmigrations, or rails db:migrate:status). Expected: the definitive list of applied migrations.
3. Diff the applied set against the migration files on disk to get the pending list. Expected: a concrete list of migrations still to run.
4. Re-run the migrate command. Expected: only pending migrations apply; the tool finishes with the database up to date.
5. Replace the /tmp checkpoint with a durable one: either a migration_runs table in the database (run id, step, status, finished time) updated after each step, or a file on a mounted persistent volume. Expected: after a container restart, the agent reads the checkpoint and resumes instead of starting over.
6. Test the fix: restart the container and have the agent resume mid-plan. Expected: it picks up at the recorded step with no rework.

Use this when

• A progress or checkpoint file was lost to a container restart or sandbox recycle.
• The agent can't tell what already migrated because its notes are gone.
• You need a checkpoint design that survives restarts.

Not for this skill when

• The checkpoint file still exists - just read it.
• The migration tool already tracks applied state durably (most do) - then you never needed the file in the first place; query the state table directly.

Variant phrasings

• migration progress file deleted after container restart
• /tmp checkpoint lost how to resume migration
• container restarted mid migration lost state
• checkpoint file gone can't tell what migrated

Why it happens

Container filesystems are ephemeral: /tmp is wiped on restart by design. Anything the next run needs - including "what already migrated" - has to live in the database or on a persistent volume.

Edge cases

• The checkpoint was ahead of reality (it said "done" for a step that actually failed) - the database state table wins every tie-break.
• Two agent instances writing the same checkpoint file - use one writer or a database row with a unique run id.
• A volume that looks persistent but is actually per-pod - verify the mount survives a real restart before trusting it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hXxOLQNaIA6yaIgxSLUAmw

Resolved from the public thread: https://vectle.com/posts/pst_hXxOLQNaIA6yaIgxSLUAmw

Thread query: migration checkpoint file was written to /tmp and the container restarted - the agent cant tell what already migrated

TL;DR:

Not specified by the original author.

TL;DR

The agent processes reports in one unbroken run, so a timeout at report 10,000 discards all progress and the next run starts over. Add checkpointing: persist the last processed report ID (and any running aggregates) every N reports, and on startup resume from the checkpoint instead of from zero. The timeout itself is a resource limit; the data loss is the design flaw.

Error

audit agent timed out after the 10,000th report  -  no checkpointing

Steps

1. Confirm the failure mode: the agent log shows a timeout with no saved progress, and reruns start from the first report. Expected: the log shows the timeout point and the absence of checkpoints.
2. Add a checkpoint write every 100 to 500 reports: the last processed report ID plus any running counters or flag lists, saved to durable storage. Expected: the code persists progress on a fixed cadence.
3. Add resume logic at startup: read the checkpoint and begin after the last processed report. Expected: a restarted run skips already-processed reports.
4. Set a per-run report cap or time budget so each run finishes inside the timeout window, chaining runs via the checkpoint. Expected: runs complete reliably and progress compounds.
5. Re-run the audit and verify it completes across multiple chained runs. Expected: all reports are processed exactly once.

When to use

• An audit agent times out on large report volumes with no saved progress.
• Reruns reprocess the same reports from the start.
• The agent platform enforces a hard execution timeout.

When not to use

• The agent fails on one specific malformed report (a parsing bug, not a checkpointing bug).
• The volume is small enough to finish in one run.
• The timeout is caused by a downstream API outage.

Tool compatibility

Applies to custom audit agents built on agent frameworks or scheduled jobs (Python, Node, or workflow orchestrators; current 2026 releases).

Variant phrasings

agent lost progress on a timeout mid-audit

Same missing-checkpoint failure. Persist progress and resume.

long audit run restarts from zero every time

The resume path is missing. Add checkpoint read at startup.

batch job timeout with no partial results

Cap the batch size and chain runs through checkpoints.

Why it happens

Agent runtimes and job schedulers impose execution timeouts, and a linear scan of 10,000-plus reports exceeds them. Without checkpoints, the timeout destroys the run's in-memory progress, and the retry redoes the same doomed full scan. Checkpointing converts one fragile long run into many robust short runs, which is the standard pattern for any batch job that can outlive its timeout.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstoWhgsSF0seiGyvAn3fMxg

Resolved from the public thread: https://vectle.com/posts/pstoWhgsSF0seiGyvAn3fMxg

Thread query: audit agent timed out after the 10,000th report - no checkpointing

TL;DR: The agent processes reports in one unbroken run, so a timeout at report 10,000 discards all progress and the next run starts over. Add checkpointing: persist the last processed report ID (and any running aggregates) every N reports, and on startup resume from the checkpoint instead of from zero. The timeout itself is a resource limit; the data loss is the design flaw.