## TL;DR
Key the politeness bucket by host (scheme plus hostname) instead of by registrable domain, so each subdomain gets its own delay budget. Honor Retry-After and 429 as instructions to slow that host down, and keep per-host concurrency caps. Throttling stops instead of escalating.

```text
agent's crawl politeness delay applied per-domain instead of per-host and got throttled
```

## Use this when
- a crawler gets throttled on sites with many subdomains
- api and www subdomains share one delay and one gets starved
- 429 responses keep coming despite a politeness delay
- you need to prove the delay was honored per host

## Not for this skill when
- the site blocks the crawler regardless of rate
- you need to go faster than the site allows
- robots rules are the actual problem

## Steps
1. Key the politeness bucket by host (scheme plus hostname), not by registrable domain, so subdomains get independent budgets. Expected: api.example.com and www.example.com no longer share one delay.
2. Set the per-host delay from the site's response behavior (Retry-After, observed latency) with a sane floor. Expected: delays match what each host tolerates.
3. Treat 429 and Retry-After as instructions to slow that host down, not as errors to retry immediately. Expected: throttling stops instead of escalating.
4. Keep a per-host request log with timestamps so you can prove the delay was honored. Expected: debugging a throttle takes minutes.
5. Cap concurrency per host separately from the global pool. Expected: one aggressive host cannot starve or flood others.

## Variant phrasings
### crawler throttled because subdomains shared one delay
Short version: bucket politeness by host so each subdomain gets its own budget.
### politeness delay applied per-domain not per-host
Short version: key the rate bucket on scheme plus hostname.
### 429s kept coming despite the crawl delay
Short version: treat Retry-After as a slow-down instruction, not a retry cue.

Compatibility: works with Scrapy, custom asyncio crawlers, or any agent loop; host keying is a small change to the delay lookup, no library pinned.

## Why it happens
The delay was keyed on the registrable domain, so ten subdomains shared a single bucket meant for one host. Some hosts got hammered while the agent believed it was being polite. The site saw per-host request rates far above the intended delay and throttled the crawler.

## Edge cases / pitfalls
- Rate limits are a signal to slow down; never rotate IPs, user agents, or endpoints to get around them.
- Some sites rate-limit by path or API key rather than host; per-host buckets do not fix those, so watch for 429 patterns that ignore host.
- A sane floor (for example one second) still applies to unknown hosts; do not start at zero delay.
- Per-host logs grow on huge crawls; aggregate to counts per minute per host after debugging.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EnkKepifaqfqndA_XSC3Ng