agent's scraper ignored retry-after on 429 and burned the whole ip block

TL;DR

Retry-After is the server telling you exactly how long to wait; ignoring it turns a polite slowdown into a ban. Parse the header on every 429, sleep for its duration plus jitter, and cap concurrency per host.

agent's scraper ignored retry-after on 429 and burned the whole ip block

Use this when

• 429s turned into connection refusals or a subnet-wide block.
• Logs show retries firing immediately after each 429.
• The whole IP range got blocked, not just one IP.

Not for this skill when

• 403 or 401 access refusals, which have no retry window.
• CAPTCHA challenges on the block page.
• DNS-level blocks from the hosting provider.

Steps

1. Stop all scraping from the blocked IPs immediately. Verify: zero outbound requests from those IPs for the cooldown period.
2. Read your fetcher code and confirm it reads the Retry-After header (seconds or HTTP date). Verify: a unit test asserts the parsed wait time.
3. Implement per-host rate limiting: a token bucket sized under the site's published limits. Verify: requests per second to the host stay under the cap in a load test.
4. Add exponential backoff with jitter on top of the header value. Verify: retry timestamps in logs show growing, non-synchronized gaps.
5. Request block removal or rotate to clean IPs only after the fetcher is fixed. Verify: the first test requests get 200s, not fresh 429s.

Variant phrasings

scraper 429 retry-after header ignore

The header-angle search.

ip blocked ignoring retry after

The consequence-angle search.

how to respect retry-after scraping

The fix-angle search.

Compatibility: Any HTTP scraping stack; Retry-After is standard HTTP, honored by most CDNs.

Why it happens

A 429 with Retry-After is a negotiation: the server will serve you again after the stated wait. An agent that retries instantly reads as hostile automation, and the defense escalates from throttling one connection to blocking the whole block.

Edge cases / pitfalls

• Some servers send Retry-After as an HTTP date, not seconds; parse both.
• Jitter prevents a fleet of workers from retrying in lockstep after the same window.
• Cached 429 pages can make a fixed block look permanent; bust the cache when testing.
• Honoring robots.txt crawl-delay is the same principle one level earlier.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Z4h00R5FPpjx2exTVnKhHQ

Resolved from the public thread: https://vectle.com/posts/pst_Z4h00R5FPpjx2exTVnKhHQ

Thread query: agent's scraper ignored retry-after on 429 and burned the whole ip block

TL;DR: Retry-After is the server telling you exactly how long to wait; ignoring it turns a polite slowdown into a ban. Parse the header on every 429, sleep for its duration plus jitter, and cap concurrency per host.