sec edgar user-agent rejected 403 bulk download failed

SEC EDGAR user agent rejected with 403

TL;DR

EDGAR returns 403 when your requests carry a missing, generic, or bot-library default user agent, because the SEC requires crawlers to identify themselves under its fair-access policy. The fix is a descriptive user agent string naming your bot, kept on every request including redirects and bulk downloads. Pair it with the 10-requests-per-second ceiling and Retry-After backoff, and the 403s clear without any other change.

The error

HTTP 403 Forbidden
Your request was rejected: user agent missing or not descriptive (SEC fair-access policy)

When this helps

• EDGAR downloads fail with 403 on a new downloader
• a library upgrade silently reset the user agent to a default
• bulk archive pulls 403 while browser visits work fine
• auditing an EDGAR pipeline for fair-access compliance

When it doesn't

• the failure is a 429 rate limit; that is a pacing problem, not identification
• you want to spoof a browser user agent to blend in; use an honest bot identifier instead
• the 403 persists with a proper user agent, then check for an IP-level block

Works with

Any HTTP client. The SEC's user-agent requirement applies to www.sec.gov and data.sec.gov alike as of 2026.

Steps

1. Reproduce with your current user agent to confirm the cause

curl -s -o /dev/null -w "HTTP %{http_code}\n" "https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]"
curl -s -o /dev/null -w "HTTP %{http_code}\n" -A "IntelBriefingBot/1.0" "https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]"

Expected: First request 403s, second returns 200. That split proves the user agent was the problem, not your IP or rate.

2. Set a descriptive user agent on the session, not per request

import requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
r = s.get("https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=[cik]&type=10-K", timeout=30)
print(r.status_code)

Expected: HTTP 200. Setting it on the session covers redirects and helper calls that a per-request header would miss.

3. Keep the rate under the fair-access ceiling

import time, requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
for i in range(3):
    r = s.get("https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]", timeout=30)
    print(i, r.status_code)
    time.sleep(1)

Expected: Three 200s. A proper user agent plus a sane rate is the whole SEC compliance story; skip either one and the 403s or 429s come back.

4. Verify bulk downloads carry the same identification

curl -s -A "IntelBriefingBot/1.0" "https://www.sec.gov/Archives/edgar/daily-index/" -o index.html -w "HTTP %{http_code}\n"

Expected: HTTP 200 on the archive index. Bulk paths enforce the same policy, so check the identifier on every code path that touches sec.gov.

Other ways people phrase this

sec edgar 403 user agent bulk download failed

Bulk paths enforce the same rule. The identifier must be on every request the bulk job makes.

edgar request rejected user agent policy

The policy wording. Descriptive bot identification is the entire fix.

403 forbidden sec.gov scraper

Check the user agent before anything else; it is the most common cause of EDGAR 403s by far.

Why it happens

The SEC asks every automated client to identify itself so it can contact abusive crawlers instead of blocking blindly. Requests with no user agent, or with a library default like python-requests, look like unidentified bots and get a 403. It is an identification check, not a judgment on your crawl rate.

Edge cases

• Redirects can drop custom headers in some clients; setting the agent on the session object avoids that.
• The policy applies per request, so one-off scripts and notebook cells need the header too, not just the production crawler.
• A descriptive agent plus 20 requests per second still 429s; identification and rate are two separate requirements.
• If a previously working agent starts 403ing, check for a library upgrade that reset the default user agent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8zMpxI2AOVvXwGrzyTtUBA