## TL;DR
Persist the crawl queue to disk before every request, and checkpoint it again the moment a 429 arrives. Then back off with exponential sleep plus Retry-After, and resume from the saved queue instead of rebuilding it. Restarting from scratch turns a temporary rate limit into wasted hours and a ban risk from refetching.

```text
scraper agent hit 429 mid-crawl and restarted the whole queue from scratch
```

## Use this when
- a crawl loses its queue after a 429 and starts over
- the agent refetches URLs it already visited
- the crawl has no saved state between restarts

## Not for this skill when
- you need proxy rotation strategy (separate concern)
- the site blocks you outright with 403 or a CAPTCHA
- you are writing a scraper from zero and just need structure

## Steps
1. Write the queue (pending plus done URL lists) to a local JSON file before starting the crawl. Verify: the file exists and contains the full seed list.
2. After each completed request, remove the URL from pending and append it to done. Verify: reopening the file shows progress moving forward.
3. On a 429, save the current queue immediately, then sleep with exponential backoff starting at 60 seconds, honoring any Retry-After header. Verify: logs show the save happened before the sleep.
4. On restart, load the saved queue and continue from pending only. Verify: no URL from the done list gets requested again.
5. Rate-limit yourself: max one request per N seconds per host, enforced in code not by luck. Verify: a dry run of 10 URLs takes the expected time.

### Tool/version compatibility
Any Python HTTP client (requests, httpx) or Playwright-based crawler; the pattern is framework-free.

## Why it happens
The agent keeps its queue in memory, so any crash or restart discards it. The 429 becomes the trigger but the missing checkpoint is the real bug.

## Edge cases / pitfalls
- Rate limits are a signal to slow down, not a wall to go around. Never add parallel workers to make up time after a 429.
- Retry-After can be given in seconds or as a date; handle both.
- A corrupted checkpoint file should fail loudly, not silently restart from zero.

## Variant phrasings
### Agent kept refetching pages after being rate limited
Short version: crawler restarts the queue on 429.
### Scraper lost all progress when it got throttled
Short version: 429 wiped the crawl queue.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-PNieCPRxqVQlt1YkN9rGg