## TL;DR
Scrapy's retry middleware does not honor `Retry-After` out of the box. Add 429 to `RETRY_HTTP_CODES` (or write a small middleware that sleeps the header's value), and set a sane `RETRY_TIMES` and download delay. If 429s keep coming after you honor the header, the site wants you slower or gone.

```text
scrapy 429 too many requests retry middleware ignoring Retry-After header
```

## Use this when
- Logs show `429 Unknown Status` retried immediately, seconds apart.
- The target sends a `Retry-After` header that your spider never waits for.
- You are crawling with permission and the fix is politeness, not volume.

## Not for this skill when
- The status is 403. That is a refusal, and retry middleware is not the tool.
- The 429 comes with a block page or CAPTCHA. Waiting does not fix a challenge.

## Steps
1. Confirm the header is present: log `response.headers.get(b"Retry-After")` on a 429. Verify: you see an integer number of seconds.
2. Add the code to retries: `RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]` in settings.py. Verify: retries still occur, now classified as retryable.
3. Install a middleware that reads the header before retrying. Minimal version: subclass `RetryMiddleware`, in `process_response` check `response.status == 429`, read the header, `time.sleep(value)` or reschedule with `dont_filter=True`. Verify: request spacing on 429s matches the header value in logs.
4. Set `RETRY_TIMES = 3` and `DOWNLOAD_DELAY = 2` as a floor. Verify: overall request rate drops and 429 frequency falls.
5. If 429s persist after honoring Retry-After twice, back off the crawl (halve CONCURRENT_REQUESTS) or stop that domain. Verify: the domain stops returning 429 for the next hour.

## Variant phrasings
### scrapy retry middleware 429
The short phrasing, lands on the same settings.
### scrapy does not wait Retry-After
Symptom phrasing, points at the custom middleware in step 3.
### scrapy too many requests immediate retry
Behavior phrasing, the diagnosis in step 1.

Compatibility: Scrapy 2.x. The default retry middleware honors nothing about timing; the header handling is always custom.

## Why it happens
The retry middleware was built for transient 5xx failures, where immediate retry is fine. A 429 is a rate signal, not a blip, and the stock middleware treats it identically: it re-queues the request with no delay. Honoring `Retry-After` is a small custom-middleware job, not a setting, which is why so many spiders miss it.

## Edge cases / pitfalls
- Some servers send `Retry-After` as an HTTP date, not seconds. Parse both.
- `AUTOTHROTTLE_ENABLED` helps with gradual overload but does not read the header either. Use both.
- Sleeping inside `process_response` blocks the whole engine. For big crawls, reschedule with a delayed re-request instead of `time.sleep`.
- A 429 that never clears after backoff is a soft ban. Continuing to crawl is the problem, not the middleware.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tyOrw6VaTjTHqPP8h02NQw