# SEC EDGAR 429 rate limit on filing downloads

## TL;DR
EDGAR 429s mean you are pulling filings faster than the SEC's fair-access policy allows, currently 10 requests per second, and the polite fix is to slow down, not to parallelize harder. Identify your bot with a descriptive user agent, stay under the rate, honor Retry-After, and lean on the SEC's bulk daily and quarterly archives so you are not hammering the API for files that change rarely. Retries with backoff clear genuine 429s; a 429 that never clears means your crawl pattern itself is the problem.

## The error
```text
HTTP 429 Too Many Requests
(filing download fails; response headers include Retry-After)
```

## When this helps
- a filing downloader starts returning 429s mid batch
- building a new EDGAR puller and picking a crawl rate
- a briefing agent fails during earnings week when filing traffic spikes
- designing retries for the SEC's full-text search or company APIs

## When it doesn't
- you want to dodge the limit with parallel workers or rotating IPs; that violates fair access and risks a block
- the failure is a 403 user-agent rejection instead, see the user-agent skill
- you need real-time streaming data EDGAR does not publish

## Works with
Works against www.sec.gov and data.sec.gov as of 2026. Any HTTP client; the 10 requests per second ceiling and user-agent policy are SEC-side.

## Steps
### 1. Confirm the rate response and read the retry hint
```bash
curl -s -D headers.txt -o filing.html -A "IntelBriefingBot/1.0" "https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]"
grep -i "retry-after\|x-rate" headers.txt || echo "no rate headers returned"
```
Expected: A 429 status plus a Retry-After value telling you how long to wait. No headers at all can mean you tripped a different throttle; back off anyway.

### 2. Drop your crawl rate under the fair-access ceiling
```python
import time, requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
for url in ["https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=[cik]&type=10-K"]:
    r = s.get(url, timeout=30)
    print(r.status_code)
    time.sleep(0.5)  # stays well under 10 requests per second
```
Expected: 200s at the slower pace. If 429s persist at one request every couple of seconds, something else in your pattern is flagged; widen the gap further.

### 3. Add backoff that respects Retry-After
```python
import time, requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
for attempt in range(5):
    r = s.get("https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]", timeout=30)
    if r.status_code != 429:
        break
    wait = int(r.headers.get("Retry-After", 10))
    print("429, waiting", wait)
    time.sleep(wait)
print("final", r.status_code)
```
Expected: The loop exits on the first non-429. Five straight 429s means stop for the day and re-check your rate tomorrow; do not widen the retry storm.

### 4. Switch bulk history pulls to the archives
```bash
curl -s "https://www.sec.gov/Archives/edgar/daily-index/" | head -20
```
Expected: The daily and quarterly index listings. Filing text that is months old never changes, so fetch each accession once, store it locally, and never re-download it.

## Other ways people phrase this
### edgar too many requests error bulk download
Same 429. Bulk history is the usual trigger; the daily and quarterly archives exist precisely so you do not need to scrape history one filing at a time.

### sec.gov download failed retry after header
The server told you how long to wait. Parse Retry-After and sleep that long before the next attempt; ignoring it is how a 429 turns into a block.

### 429 on sec full text search api
The search API shares the same throttling. Shrink the result window, cache aggressively, and back off the same way.

## Why it happens
EDGAR is a free public service and the SEC throttles it to keep it usable for everyone. Ten requests per second per client is the published ceiling; bursts, parallel workers, or re-downloading the same filings trip it. The limit protects the archive, and the SEC blocks clients that treat it as a suggestion.

## Edge cases
- The rate limit is per client behavior, not per API key; spreading load across workers to evade it is exactly what gets blocked.
- Retry-After is advisory in seconds; some clients see minutes during earnings-week load, so parse it instead of hardcoding.
- Full-text search has its own limits separate from filing downloads; throttle the two paths independently.
- A 429 that appears instantly on the first request of the day usually means a cached block from yesterday; wait it out rather than hammering.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qV66xEbJ2b9F1KYkagD3SQ