audit agent timed out after the 10,000th report - no checkpointing

TL;DR

The agent processes reports in one unbroken run, so a timeout at report 10,000 discards all progress and the next run starts over. Add checkpointing: persist the last processed report ID (and any running aggregates) every N reports, and on startup resume from the checkpoint instead of from zero. The timeout itself is a resource limit; the data loss is the design flaw.

Error

audit agent timed out after the 10,000th report  -  no checkpointing

Steps

1. Confirm the failure mode: the agent log shows a timeout with no saved progress, and reruns start from the first report. Expected: the log shows the timeout point and the absence of checkpoints.
2. Add a checkpoint write every 100 to 500 reports: the last processed report ID plus any running counters or flag lists, saved to durable storage. Expected: the code persists progress on a fixed cadence.
3. Add resume logic at startup: read the checkpoint and begin after the last processed report. Expected: a restarted run skips already-processed reports.
4. Set a per-run report cap or time budget so each run finishes inside the timeout window, chaining runs via the checkpoint. Expected: runs complete reliably and progress compounds.
5. Re-run the audit and verify it completes across multiple chained runs. Expected: all reports are processed exactly once.

When to use

• An audit agent times out on large report volumes with no saved progress.
• Reruns reprocess the same reports from the start.
• The agent platform enforces a hard execution timeout.

When not to use

• The agent fails on one specific malformed report (a parsing bug, not a checkpointing bug).
• The volume is small enough to finish in one run.
• The timeout is caused by a downstream API outage.

Tool compatibility

Applies to custom audit agents built on agent frameworks or scheduled jobs (Python, Node, or workflow orchestrators; current 2026 releases).

Variant phrasings

agent lost progress on a timeout mid-audit

Same missing-checkpoint failure. Persist progress and resume.

long audit run restarts from zero every time

The resume path is missing. Add checkpoint read at startup.

batch job timeout with no partial results

Cap the batch size and chain runs through checkpoints.

Why it happens

Agent runtimes and job schedulers impose execution timeouts, and a linear scan of 10,000-plus reports exceeds them. Without checkpoints, the timeout destroys the run's in-memory progress, and the retry redoes the same doomed full scan. Checkpointing converts one fragile long run into many robust short runs, which is the standard pattern for any batch job that can outlive its timeout.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstoWhgsSF0seiGyvAn3fMxg