## TL;DR
A 429 mid-crawl is normal backpressure, not a failure: the agent should back off, persist its position (cursor, page number, last id) continuously, and resume from the cursor instead of restarting. The fix is a durable cursor plus polite retry, not a faster crawl.

```text
scraper agent hit 429 mid-crawl: resume with cursor
```

## Use this when
- A scraper agent dies on 429 halfway through
- Every retry restarts the crawl from the beginning
- 429s are treated as fatal errors

## Not for this skill when
- The IP is banned (thats a ban, not throttling)
- A login wall blocks (thats auth)
- Pages parse wrong (thats the parser)

## Steps

1. Persist the cursor after every page, not at the end:

```python
# after each successful page:
state["last_id"] = page[-1]["id"]
save_state(state)  # write to disk, not just memory
```
Expected output: a crash at page 500 leaves the cursor at 499. In-memory cursors die with the process; that is the whole bug.

2. On startup, resume from the cursor:

```python
state = load_state()
start_from = state.get("last_id")  # None means start at the beginning
```
Expected output: reruns continue where the last run stopped. The agent must do this unconditionally, not only "if it was throttled."

3. Handle 429 with backoff and Retry-After:

```python
import time
if response.status_code == 429:
    wait = int(response.headers.get("Retry-After", 60))
    time.sleep(wait)
    continue  # retry the same page, cursor unchanged
```
Expected output: the crawl slows down instead of dying. Honor the server's asked wait; it is telling you exactly how long.

4. Slow the baseline crawl rate so 429s are rare:

```python
# aim under the documented rate limit with margin
time.sleep(1.0)  # tune to the site's limit
```
Expected output: fewer 429s overall. A crawl that constantly rides the limit is one hiccup away from a ban.

## Variant phrasings

### 429s even at a slow rate
The limit may be per API key, per IP across all your agents, or bursty. Check what the limit scopes to before tuning further.

### cursor resume duplicates the last page
Resume is at-least-once by nature. Dedupe on the record id downstream; that is expected, not a bug.

## Why it happens
Scraper agents are usually written as straight-line loops: fetch pages 1 to N, crash on any error. A 429 is the server enforcing its rate limit, which is routine at crawl scale. Without a persisted cursor, every failure restarts from zero, which re-hits the rate limit faster (re-requesting the same pages), turning one 429 into a permanent failure loop.

## Edge cases
- Some sites 429 by IP and ban by pattern; if 429s come with captchas, stop and reassess rather than backing off forever.
- Cursor formats differ (page numbers, opaque cursor strings, timestamps); persist whatever the site's pagination actually uses.
- If the crawl target changes mid-crawl (new items inserted at the front), a positional cursor skips items; prefer id-based cursors.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fxz2gHC1axJ-4IQQ-Au4lg