# SEC EDGAR user agent rejected with 403

## TL;DR
EDGAR returns 403 when your requests carry a missing, generic, or bot-library default user agent, because the SEC requires crawlers to identify themselves under its fair-access policy. The fix is a descriptive user agent string naming your bot, kept on every request including redirects and bulk downloads. Pair it with the 10-requests-per-second ceiling and Retry-After backoff, and the 403s clear without any other change.

## The error
```text
HTTP 403 Forbidden
Your request was rejected: user agent missing or not descriptive (SEC fair-access policy)
```

## When this helps
- EDGAR downloads fail with 403 on a new downloader
- a library upgrade silently reset the user agent to a default
- bulk archive pulls 403 while browser visits work fine
- auditing an EDGAR pipeline for fair-access compliance

## When it doesn't
- the failure is a 429 rate limit; that is a pacing problem, not identification
- you want to spoof a browser user agent to blend in; use an honest bot identifier instead
- the 403 persists with a proper user agent, then check for an IP-level block

## Works with
Any HTTP client. The SEC's user-agent requirement applies to www.sec.gov and data.sec.gov alike as of 2026.

## Steps
### 1. Reproduce with your current user agent to confirm the cause
```bash
curl -s -o /dev/null -w "HTTP %{http_code}\n" "https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]"
curl -s -o /dev/null -w "HTTP %{http_code}\n" -A "IntelBriefingBot/1.0" "https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]"
```
Expected: First request 403s, second returns 200. That split proves the user agent was the problem, not your IP or rate.

### 2. Set a descriptive user agent on the session, not per request
```python
import requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
r = s.get("https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=[cik]&type=10-K", timeout=30)
print(r.status_code)
```
Expected: HTTP 200. Setting it on the session covers redirects and helper calls that a per-request header would miss.

### 3. Keep the rate under the fair-access ceiling
```python
import time, requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
for i in range(3):
    r = s.get("https://www.sec.gov/Archives/edgar/data/[cik]/[accession]/[doc]", timeout=30)
    print(i, r.status_code)
    time.sleep(1)
```
Expected: Three 200s. A proper user agent plus a sane rate is the whole SEC compliance story; skip either one and the 403s or 429s come back.

### 4. Verify bulk downloads carry the same identification
```bash
curl -s -A "IntelBriefingBot/1.0" "https://www.sec.gov/Archives/edgar/daily-index/" -o index.html -w "HTTP %{http_code}\n"
```
Expected: HTTP 200 on the archive index. Bulk paths enforce the same policy, so check the identifier on every code path that touches sec.gov.

## Other ways people phrase this
### sec edgar 403 user agent bulk download failed
Bulk paths enforce the same rule. The identifier must be on every request the bulk job makes.

### edgar request rejected user agent policy
The policy wording. Descriptive bot identification is the entire fix.

### 403 forbidden sec.gov scraper
Check the user agent before anything else; it is the most common cause of EDGAR 403s by far.

## Why it happens
The SEC asks every automated client to identify itself so it can contact abusive crawlers instead of blocking blindly. Requests with no user agent, or with a library default like python-requests, look like unidentified bots and get a 403. It is an identification check, not a judgment on your crawl rate.

## Edge cases
- Redirects can drop custom headers in some clients; setting the agent on the session object avoids that.
- The policy applies per request, so one-off scripts and notebook cells need the header too, not just the production crawler.
- A descriptive agent plus 20 requests per second still 429s; identification and rate are two separate requirements.
- If a previously working agent starts 403ing, check for a library upgrade that reset the default user agent.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8zMpxI2AOVvXwGrzyTtUBA