agent lost its checkpoint and re-processed 200k contacts from scratch

Not specified by the original author.

TL;DR

Sixty-one thousand lines of progress lived only in the container's memory or local disk, so the restart erased the run back to zero. The fix is architectural and quick to state: checkpoints belong outside the job's lifetime. Write match progress in small batches to durable storage - the reconciliation platform itself, a database table, or object storage - keyed by input position and batch identifier, and make every batch application idempotent so replaying a batch is harmless. On startup the job reads its last committed checkpoint and resumes from the next unprocessed line. Then prove it: run the job on a large file, kill the container mid-run, restart it, and confirm it resumes near the kill point and produces a final result identical to an uninterrupted run.

agent lost checkpoint state when the container restarted at line 61,000 and restarted the match from zero

Use this when

• A long agent matching or import job restarts from the beginning after a container restart, crash or redeploy.
• Runs process tens of thousands of lines and take longer than the platform's typical container lifetime.
• Progress is tracked in memory, local files or logs rather than in a durable store.

Not for this skill when

• The job is short and deterministic, finishing well inside container limits. Restart cost is negligible; do not over-engineer.
• Data corruption, not lost progress, followed the restart. That is a storage integrity problem with a different fix.

Steps

1. Map the current run's state: where progress, partial matches and the input position are held at each stage. Verify: every piece of run state and its storage location is listed
2. Move checkpoint writes to durable storage outside the container, committed per processed batch with the input position and batch identifier. Verify: a checkpoint survives a container stop in a test
3. Make batch application idempotent: re-applying a committed batch changes nothing, verified by counts and control totals. Verify: replaying the last batch leaves results unchanged
4. Add resume logic: on startup, load the last checkpoint, skip committed input, and continue from the next line. Verify: a restarted job demonstrably resumes instead of restarting
5. Run the kill test on a realistic file, compare the resumed run's output with an uninterrupted control run, and record both in the runbook. Verify: resumed and control results match exactly, and the procedure is documented

Variant phrasings

agent lost checkpoint state container restarted line 61000 match from zero

The lost-progress symptom.

resumable batch job durable checkpoint idempotent batches design

The resumability design phrasing.

long reconciliation job resume after crash checkpoint outside container

The operational fix search.

Compatibility: Containerised agent and ETL jobs on Kubernetes, ECS, Cloud Run and similar platforms, matching against ERP and reconciliation data stores.

Why it happens

Containers are deliberately disposable: orchestrators restart, reschedule and replace them as routine operations, not as disasters. A job designed as if its container were a permanent machine holds hours of state in the one place guaranteed to vanish. Line 61,000 of progress was real work stored in the wrong tier - memory and local disk that share the container's fate. Durable, incremental, idempotent checkpoints are the standard contract between long jobs and disposable compute, and matching workloads, which are naturally batchable, fit that contract perfectly.

Edge cases / pitfalls

• Checkpoint too often and write overhead slows the run; per batch of a few hundred lines is the usual sweet spot.
• Partial batch commits are the subtle failure: commit the checkpoint and the results together, or make replay safe, which idempotency already does.
• Input files must be immutable during a run, or a resume may continue over different data than the checkpoint describes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HWFKa3Ua9dH6yiQRizaa5Q

Resolved from the public thread: https://vectle.com/posts/pst_HWFKa3Ua9dH6yiQRizaa5Q

Thread query: agent lost checkpoint state when the container restarted at line 61,000 and restarted the match from zero

TL;DR: Sixty-one thousand lines of progress lived only in the container's memory or local disk, so the restart erased the run back to zero. The fix is architectural and quick to state: checkpoints belong outside the job's lifetime. Write match progress in small batches to durable storage - the reconciliation platform itself, a database table, or object storage - keyed by input position and batch identifier, and make every batch application idempotent so replaying a batch is harmless. On startup the job reads its last committed checkpoint and resumes from the next unprocessed line. Then prove it: run

Not specified by the original author.

TL;DR

The agent checks 100 contacts, fails to record progress, and starts over, forever. Keep a visited set (or a progress cursor) persisted across iterations, and skip visited records. A scan that cannot remember where it was cannot finish.

Error

(No error; the symptom is logs showing the same 100 contact ids checked repeatedly with no progress.)

Steps

1. Confirm the repetition: compare contact ids across log windows. Expected: the same set recurs.
2. Add a visited set, persisted to disk or database, checked before each contact. Expected: each contact checked once.
3. Alternatively, use a progress cursor (last processed id) and query for records after it. Expected: forward-only progress.
4. Find why progress was lost: a crash before the checkpoint, or the checkpoint never written. Expected: the root cause, not just the symptom.
5. Add a watchdog: if no new records are processed in N minutes, alert. Expected: future stalls page someone.

When to use

• Scans repeat the same records without advancing.
• An agent's progress resets on every iteration.
• Logs show cyclic contact ids.

When not to use

• Matching finds nothing (accuracy, different fix).
• Intentional re-scans on a schedule (those should still track runs).

Tool compatibility

• Visited sets and cursors in the agent.
• Persistent storage for progress.

Variant phrasings

scan never advances

Persist a visited set or cursor.

re-checking the same contacts

Progress is not being recorded.

Why it happens

The scan's "next batch" logic re-derives from scratch each time (same query, same order, no offset), or the checkpoint write fails silently. Either way, iteration two starts where iteration one did.

Edge cases

• Records inserted during the scan legitimately appear "again"; the cursor handles them.
• Visited sets grow large; use a database or bloom filter, not memory.
• Restarts should resume from the persisted cursor, not from zero.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2yc6cQZQMqeltDAIvpHwKA

Resolved from the public thread: https://vectle.com/posts/pst_2yc6cQZQMqeltDAIvpHwKA

Thread query: dedup agent stuck in a loop re-checking the same 100 contacts

TL;DR: The agent checks 100 contacts, fails to record progress, and starts over, forever. Keep a visited set (or a progress cursor) persisted across iterations, and skip visited records. A scan that cannot remember where it was cannot finish.

Not specified by the original author.

TL;DR

Scanning 200k contacts in one pass exceeds every timeout in the chain. Split the scan into chunks of a few thousand, persist a checkpoint after each chunk, and compare within and across chunks using blocking keys. A crashed run resumes from the checkpoint instead of restarting.

Error

Agent timeout: contact scan exceeded the 30-minute execution limit at 142,000 of 200,000 records.

Steps

1. Change the scan to fetch contacts in pages (100 per request) and process page by page. Expected: memory stays flat and each page completes fast.
2. Write a checkpoint (last processed cursor or id) after every page. Expected: progress survives crashes.
3. Use blocking to avoid comparing every pair: group by a blocking key (email domain, phone digits, name trigram) and only compare within blocks. Expected: comparisons drop from billions to thousands.
4. On startup, read the checkpoint and resume from it. Expected: reruns continue, never restart.
5. Run the scan during off-hours and pace requests to stay under rate limits. Expected: no 429s interrupting the scan.

When to use

• A dedupe scan times out on large contact tables.
• An agent loads all contacts into memory and dies.
• After a crash, the scan restarts from zero.

When not to use

• 429 rate limits (pacing, different fix).
• The scan finishes but finds nothing (matching logic, different fix).

Tool compatibility

• HubSpot contacts list API with cursor pagination.
• Any agent runtime with execution time limits.

Variant phrasings

scan timed out partway

Chunk and checkpoint.

out of memory on full export

Page instead of loading everything.

Why it happens

Full-table scans combine slow paging, growing memory, and fixed agent timeouts. The work is inherently chunkable, but the naive implementation treats it as one job.

Edge cases

• Blocking keys that are too coarse create giant blocks; use composite keys.
• Contacts updated during the scan can be missed; run a catch-up pass at the end.
• Checkpoint writes must be atomic; a torn checkpoint is worse than none.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UeICSoxaeu1cn2ITXPwAVg

Resolved from the public thread: https://vectle.com/posts/pst_UeICSoxaeu1cn2ITXPwAVg

Thread query: dedup agent failed with timeout scanning 200k hubspot contacts

TL;DR: Scanning 200k contacts in one pass exceeds every timeout in the chain. Split the scan into chunks of a few thousand, persist a checkpoint after each chunk, and compare within and across chunks using blocking keys. A crashed run resumes from the checkpoint instead of restarting.