agent's crawl politeness delay applied per-domain instead of per-host and got throttled

TL;DR

Key the politeness bucket by host (scheme plus hostname) instead of by registrable domain, so each subdomain gets its own delay budget. Honor Retry-After and 429 as instructions to slow that host down, and keep per-host concurrency caps. Throttling stops instead of escalating.

agent's crawl politeness delay applied per-domain instead of per-host and got throttled

Use this when

• a crawler gets throttled on sites with many subdomains
• api and www subdomains share one delay and one gets starved
• 429 responses keep coming despite a politeness delay
• you need to prove the delay was honored per host

Not for this skill when

• the site blocks the crawler regardless of rate
• you need to go faster than the site allows
• robots rules are the actual problem

Steps

1. Key the politeness bucket by host (scheme plus hostname), not by registrable domain, so subdomains get independent budgets. Expected: api.example.com and www.example.com no longer share one delay.
2. Set the per-host delay from the site's response behavior (Retry-After, observed latency) with a sane floor. Expected: delays match what each host tolerates.
3. Treat 429 and Retry-After as instructions to slow that host down, not as errors to retry immediately. Expected: throttling stops instead of escalating.
4. Keep a per-host request log with timestamps so you can prove the delay was honored. Expected: debugging a throttle takes minutes.
5. Cap concurrency per host separately from the global pool. Expected: one aggressive host cannot starve or flood others.

Variant phrasings

crawler throttled because subdomains shared one delay

Short version: bucket politeness by host so each subdomain gets its own budget.

politeness delay applied per-domain not per-host

Short version: key the rate bucket on scheme plus hostname.

429s kept coming despite the crawl delay

Short version: treat Retry-After as a slow-down instruction, not a retry cue.

Compatibility: works with Scrapy, custom asyncio crawlers, or any agent loop; host keying is a small change to the delay lookup, no library pinned.

Why it happens

The delay was keyed on the registrable domain, so ten subdomains shared a single bucket meant for one host. Some hosts got hammered while the agent believed it was being polite. The site saw per-host request rates far above the intended delay and throttled the crawler.

Edge cases / pitfalls

• Rate limits are a signal to slow down; never rotate IPs, user agents, or endpoints to get around them.
• Some sites rate-limit by path or API key rather than host; per-host buckets do not fix those, so watch for 429 patterns that ignore host.
• A sane floor (for example one second) still applies to unknown hosts; do not start at zero delay.
• Per-host logs grow on huge crawls; aggregate to counts per minute per host after debugging.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEnkKepifaqfqndAXSC3Ng