dedup agent failed with timeout scanning 200k hubspot contacts
Fixes a dedup agent timing out on a 200k-contact scan: chunk the scan, checkpoint progress, and compare in batches. Use when full-table scans die partway. Not for API rate limits.
TL;DR
Scanning 200k contacts in one pass exceeds every timeout in the chain. Split the scan into chunks of a few thousand, persist a checkpoint after each chunk, and compare within and across chunks using blocking keys. A crashed run resumes from the checkpoint instead of restarting.
Error
Agent timeout: contact scan exceeded the 30-minute execution limit at 142,000 of 200,000 records.Steps
- Change the scan to fetch contacts in pages (100 per request) and process page by page. Expected: memory stays flat and each page completes fast.
- Write a checkpoint (last processed cursor or id) after every page. Expected: progress survives crashes.
- Use blocking to avoid comparing every pair: group by a blocking key (email domain, phone digits, name trigram) and only compare within blocks. Expected: comparisons drop from billions to thousands.
- On startup, read the checkpoint and resume from it. Expected: reruns continue, never restart.
- Run the scan during off-hours and pace requests to stay under rate limits. Expected: no 429s interrupting the scan.
When to use
- A dedupe scan times out on large contact tables.
- An agent loads all contacts into memory and dies.
- After a crash, the scan restarts from zero.
When not to use
- 429 rate limits (pacing, different fix).
- The scan finishes but finds nothing (matching logic, different fix).
Tool compatibility
- HubSpot contacts list API with cursor pagination.
- Any agent runtime with execution time limits.
Variant phrasings
scan timed out partway
Chunk and checkpoint.
out of memory on full export
Page instead of loading everything.
Why it happens
Full-table scans combine slow paging, growing memory, and fixed agent timeouts. The work is inherently chunkable, but the naive implementation treats it as one job.
Edge cases
- Blocking keys that are too coarse create giant blocks; use composite keys.
- Contacts updated during the scan can be missed; run a catch-up pass at the end.
- Checkpoint writes must be atomic; a torn checkpoint is worse than none.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_UeICSoxaeu1cn2ITXPwAVg