## TL;DR

Scanning 200k contacts in one pass exceeds every timeout in the chain. Split the scan into chunks of a few thousand, persist a checkpoint after each chunk, and compare within and across chunks using blocking keys. A crashed run resumes from the checkpoint instead of restarting.

## Error

```text
Agent timeout: contact scan exceeded the 30-minute execution limit at 142,000 of 200,000 records.
```

## Steps

1. Change the scan to fetch contacts in pages (100 per request) and process page by page. Expected: memory stays flat and each page completes fast.
2. Write a checkpoint (last processed cursor or id) after every page. Expected: progress survives crashes.
3. Use blocking to avoid comparing every pair: group by a blocking key (email domain, phone digits, name trigram) and only compare within blocks. Expected: comparisons drop from billions to thousands.
4. On startup, read the checkpoint and resume from it. Expected: reruns continue, never restart.
5. Run the scan during off-hours and pace requests to stay under rate limits. Expected: no 429s interrupting the scan.

## When to use

- A dedupe scan times out on large contact tables.
- An agent loads all contacts into memory and dies.
- After a crash, the scan restarts from zero.

## When not to use

- 429 rate limits (pacing, different fix).
- The scan finishes but finds nothing (matching logic, different fix).

## Tool compatibility

- HubSpot contacts list API with cursor pagination.
- Any agent runtime with execution time limits.

## Variant phrasings

### scan timed out partway

Chunk and checkpoint.

### out of memory on full export

Page instead of loading everything.

## Why it happens

Full-table scans combine slow paging, growing memory, and fixed agent timeouts. The work is inherently chunkable, but the naive implementation treats it as one job.

## Edge cases

- Blocking keys that are too coarse create giant blocks; use composite keys.
- Contacts updated during the scan can be missed; run a catch-up pass at the end.
- Checkpoint writes must be atomic; a torn checkpoint is worse than none.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UeICSoxaeu1cn2ITXPwAVg