scraper agent hit 429 mid-crawl and restarted the whole queue from scratch
Teaches a scraping agent to checkpoint its URL queue and resume after a 429 instead of restarting. Use it when a rate limit wipes crawl progress or the queue restarts from zero. Not for writing new scrapers, proxy rotation, or evading rate limits.
TL;DR
Persist the crawl queue to disk before every request, and checkpoint it again the moment a 429 arrives. Then back off with exponential sleep plus Retry-After, and resume from the saved queue instead of rebuilding it. Restarting from scratch turns a temporary rate limit into wasted hours and a ban risk from refetching.
scraper agent hit 429 mid-crawl and restarted the whole queue from scratchUse this when
- a crawl loses its queue after a 429 and starts over
- the agent refetches URLs it already visited
- the crawl has no saved state between restarts
Not for this skill when
- you need proxy rotation strategy (separate concern)
- the site blocks you outright with 403 or a CAPTCHA
- you are writing a scraper from zero and just need structure
Steps
- Write the queue (pending plus done URL lists) to a local JSON file before starting the crawl. Verify: the file exists and contains the full seed list.
- After each completed request, remove the URL from pending and append it to done. Verify: reopening the file shows progress moving forward.
- On a 429, save the current queue immediately, then sleep with exponential backoff starting at 60 seconds, honoring any Retry-After header. Verify: logs show the save happened before the sleep.
- On restart, load the saved queue and continue from pending only. Verify: no URL from the done list gets requested again.
- Rate-limit yourself: max one request per N seconds per host, enforced in code not by luck. Verify: a dry run of 10 URLs takes the expected time.
Tool/version compatibility
Any Python HTTP client (requests, httpx) or Playwright-based crawler; the pattern is framework-free.
Why it happens
The agent keeps its queue in memory, so any crash or restart discards it. The 429 becomes the trigger but the missing checkpoint is the real bug.
Edge cases / pitfalls
- Rate limits are a signal to slow down, not a wall to go around. Never add parallel workers to make up time after a 429.
- Retry-After can be given in seconds or as a date; handle both.
- A corrupted checkpoint file should fail loudly, not silently restart from zero.
Variant phrasings
Agent kept refetching pages after being rate limited
Short version: crawler restarts the queue on 429.
Scraper lost all progress when it got throttled
Short version: 429 wiped the crawl queue.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_-PNieCPRxqVQlt1YkN9rGg