scrapy 429 too many requests retry middleware ignoring Retry-After header

TL;DR

Scrapy's retry middleware does not honor Retry-After out of the box. Add 429 to RETRY_HTTP_CODES (or write a small middleware that sleeps the header's value), and set a sane RETRY_TIMES and download delay. If 429s keep coming after you honor the header, the site wants you slower or gone.

scrapy 429 too many requests retry middleware ignoring Retry-After header

Use this when

• Logs show 429 Unknown Status retried immediately, seconds apart.
• The target sends a Retry-After header that your spider never waits for.
• You are crawling with permission and the fix is politeness, not volume.

Not for this skill when

• The status is 403. That is a refusal, and retry middleware is not the tool.
• The 429 comes with a block page or CAPTCHA. Waiting does not fix a challenge.

Steps

1. Confirm the header is present: log response.headers.get(b"Retry-After") on a 429. Verify: you see an integer number of seconds.
2. Add the code to retries: RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429] in settings.py. Verify: retries still occur, now classified as retryable.
3. Install a middleware that reads the header before retrying. Minimal version: subclass RetryMiddleware, in process_response check response.status == 429, read the header, time.sleep(value) or reschedule with dont_filter=True. Verify: request spacing on 429s matches the header value in logs.
4. Set RETRY_TIMES = 3 and DOWNLOAD_DELAY = 2 as a floor. Verify: overall request rate drops and 429 frequency falls.
5. If 429s persist after honoring Retry-After twice, back off the crawl (halve CONCURRENT_REQUESTS) or stop that domain. Verify: the domain stops returning 429 for the next hour.

Variant phrasings

scrapy retry middleware 429

The short phrasing, lands on the same settings.

scrapy does not wait Retry-After

Symptom phrasing, points at the custom middleware in step 3.

scrapy too many requests immediate retry

Behavior phrasing, the diagnosis in step 1.

Compatibility: Scrapy 2.x. The default retry middleware honors nothing about timing; the header handling is always custom.

Why it happens

The retry middleware was built for transient 5xx failures, where immediate retry is fine. A 429 is a rate signal, not a blip, and the stock middleware treats it identically: it re-queues the request with no delay. Honoring Retry-After is a small custom-middleware job, not a setting, which is why so many spiders miss it.

Edge cases / pitfalls

• Some servers send Retry-After as an HTTP date, not seconds. Parse both.
• AUTOTHROTTLE_ENABLED helps with gradual overload but does not read the header either. Use both.
• Sleeping inside process_response blocks the whole engine. For big crawls, reschedule with a delayed re-request instead of time.sleep.
• A 429 that never clears after backoff is a soft ban. Continuing to crawl is the problem, not the middleware.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tyOrw6VaTjTHqPP8h02NQw