sec edgar rate limit 429 filing download failed

SEC EDGAR 429 rate limit on filing downloads

TL;DR

EDGAR 429s mean you are pulling filings faster than the SEC's fair-access policy allows, currently 10 requests per second, and the polite fix is to slow down, not to parallelize harder. Identify your bot with a descriptive user agent, stay under the rate, honor Retry-After, and lean on the SEC's bulk daily and quarterly archives so you are not hammering the API for files that change rarely. Retries with backoff clear genuine 429s; a 429 that never clears means your crawl pattern itself is the problem.

The error

HTTP 429 Too Many Requests
(filing download fails; response headers include Retry-After)

When this helps

• a filing downloader starts returning 429s mid batch
• building a new EDGAR puller and picking a crawl rate
• a briefing agent fails during earnings week when filing traffic spikes
• designing retries for the SEC's full-text search or company APIs

When it doesn't

• you want to dodge the limit with parallel workers or rotating IPs; that violates fair access and risks a block
• the failure is a 403 user-agent rejection instead, see the user-agent skill
• you need real-time streaming data EDGAR does not publish

Works with

Works against www.sec.gov and data.sec.gov as of 2026. Any HTTP client; the 10 requests per second ceiling and user-agent policy are SEC-side.

Steps

1. Confirm the rate response and read the retry hint

curl -s -D headers.txt -o filing.html -A "IntelBriefingBot/1.0" "https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]"
grep -i "retry-after\|x-rate" headers.txt || echo "no rate headers returned"

Expected: A 429 status plus a Retry-After value telling you how long to wait. No headers at all can mean you tripped a different throttle; back off anyway.

2. Drop your crawl rate under the fair-access ceiling

import time, requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
for url in ["https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=[cik]&type=10-K"]:
    r = s.get(url, timeout=30)
    print(r.status_code)
    time.sleep(0.5)  # stays well under 10 requests per second

Expected: 200s at the slower pace. If 429s persist at one request every couple of seconds, something else in your pattern is flagged; widen the gap further.

3. Add backoff that respects Retry-After

import time, requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
for attempt in range(5):
    r = s.get("https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]", timeout=30)
    if r.status_code != 429:
        break
    wait = int(r.headers.get("Retry-After", 10))
    print("429, waiting", wait)
    time.sleep(wait)
print("final", r.status_code)

Expected: The loop exits on the first non-429. Five straight 429s means stop for the day and re-check your rate tomorrow; do not widen the retry storm.

4. Switch bulk history pulls to the archives

curl -s "https://www.sec.gov/Archives/edgar/daily-index/" | head -20

Expected: The daily and quarterly index listings. Filing text that is months old never changes, so fetch each accession once, store it locally, and never re-download it.

Other ways people phrase this

edgar too many requests error bulk download

Same 429. Bulk history is the usual trigger; the daily and quarterly archives exist precisely so you do not need to scrape history one filing at a time.

sec.gov download failed retry after header

The server told you how long to wait. Parse Retry-After and sleep that long before the next attempt; ignoring it is how a 429 turns into a block.

429 on sec full text search api

The search API shares the same throttling. Shrink the result window, cache aggressively, and back off the same way.

Why it happens

EDGAR is a free public service and the SEC throttles it to keep it usable for everyone. Ten requests per second per client is the published ceiling; bursts, parallel workers, or re-downloading the same filings trip it. The limit protects the archive, and the SEC blocks clients that treat it as a suggestion.

Edge cases

• The rate limit is per client behavior, not per API key; spreading load across workers to evade it is exactly what gets blocked.
• Retry-After is advisory in seconds; some clients see minutes during earnings-week load, so parse it instead of hardcoding.
• Full-text search has its own limits separate from filing downloads; throttle the two paths independently.
• A 429 that appears instantly on the first request of the day usually means a cached block from yesterday; wait it out rather than hammering.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qV66xEbJ2b9F1KYkagD3SQ