Crunchbase enrichment agent assumed 50 req/sec but the key was throttled to 1 req/sec - 10k queued jobs all failed

TL;DR: Read the real rate limit from the API docs and response headers, then throttle the client to 1 request per second with a proper queue. The 10k failed jobs failed because the client assumed a limit 50x the real one. After re-queueing at the correct pace, the jobs complete.

Crunchbase enrichment agent assumed 50 req/sec but the key was throttled to 1 req/sec  -  10k queued jobs all failed

1. Check the documented limit for the key's plan and confirm it in a response header on a test call. Expected: the real limit is 1 request per second.
2. Stop the failed queue and count how many jobs failed versus completed. Expected: nearly all 10k failed with throttle errors.
3. Add a rate limiter to the client: one request per second max, with a small safety margin under it. Expected: a 10-request test batch takes about 10 seconds and all succeed.
4. Re-queue the failed jobs and monitor the first hundred. Expected: success rate near 100 percent with no throttle errors.

Use this when

• A bulk enrichment queue fails en masse with rate-limit or throttle errors
• The client was built against an assumed rate limit that was never verified
• Re-running the queue at the assumed pace fails the same way

Not for this skill when

• Jobs fail with 401 or 403, which means the key is invalid or the plan lapsed
• The throttle errors persist even at 1 request per second (the key may be further restricted, check the plan)
• The failures are data errors on specific companies, not rate errors

Variant phrasings

Crunchbase API throttling enrichment jobs at 1 req per second

Bulk Crunchbase queue failed, assumed wrong rate limit

Crunchbase key throttled, 10k jobs failed

Why it happens

Rate limits vary by plan, and the number a developer remembers from the docs is often the top-tier figure. A client hardcoded to 50 per second against a key limited to 1 per second burns through the quota instantly, and every queued job fails. The limit is per key, so no amount of retrying at the same pace helps.

Edge cases

• Failed jobs that partially wrote results before failing: dedupe by company ID on re-queue so nothing gets enriched twice
• Burst allowances: some plans allow short bursts above the sustained rate, but do not build the steady-state pace on the burst figure
• Plan upgrades change the limit: re-read the documented limit after any billing change instead of trusting the old number

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6O49iKHiuf5qp2DOxpG18g