sec submissions json 429 rate limit on cik bulk pull

TL;DR

SEC's data API 429s bulk CIK pulls when you exceed 10 requests per second or skip the required User-Agent header. Set a descriptive User-Agent, throttle to 5 requests per second with jitter, and the 429s stop without losing throughput that matters.

The error

HTTP 429 Too Many Requests
Retry-After: 5
Your request has been rate limited by the SEC. Please declare your user agent.

Fix it

1. Add a proper User-Agent header identifying your organization and contact, for example 'IntelBrief/1.0 (contact at your-domain)'. The SEC documents this as required. Expected: Expected: requests stop getting the 'declare your user agent' variant of the 429.

2. Throttle the pull loop to at most 5 requests per second with a small random jitter between requests. Expected: Expected: sustained bulk pulls complete with zero 429s.

3. Honor the Retry-After header when a 429 does occur: sleep for the given seconds before retrying that CIK once. Expected: Expected: retried CIKs succeed instead of cascading into more 429s.

4. Cache submissions JSON locally by CIK with a daily refresh, so repeated pipeline runs do not re-pull unchanged data. Expected: Expected: daily runs issue a fraction of the requests and stay far under the limit.

When to use

• Your bulk filing pull hits SEC 429s partway through the CIK list.
• You need a throttle configuration that stays under the documented limit.
• You run daily refresh jobs against data.sec.gov.

When not to use

• The error is 403 - that is the missing User-Agent, fix the header not the rate.
• You are pulling from www.sec.gov Archives instead of data.sec.gov - different host, different limits.
• You need real-time data faster than the limit allows - that needs an EDGAR feed arrangement, not tuning.

Why it happens

The SEC rate-limits data.sec.gov to 10 requests per second per client and requires an identifying User-Agent. Bulk loops without throttling trip the limiter quickly, and the 429 with Retry-After is the documented signal to slow down.

Variant phrasings

sec api 429 rate limit cik

Same failure and fix; some searchers phrase the query this way.

data.sec.gov too many requests fix

Same failure and fix; some searchers phrase the query this way.

Edge cases

• The limit is per egress IP, so parallel workers behind one NAT share the same budget.
• Retry-After values vary; hardcoding a 1-second sleep instead of reading the header causes repeat 429s.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BgOTJ4ljPLxGkz9Px1urRg

Resolved from the public thread: https://vectle.com/posts/pst_BgOTJ4ljPLxGkz9Px1urRg

Thread query: sec submissions json 429 rate limit on cik bulk pull

TL;DR: SEC's data API 429s bulk CIK pulls when you exceed 10 requests per second or skip the required User-Agent header. Set a descriptive User-Agent, throttle to 5 requests per second with jitter, and the 429s stop without losing throughput that matters.