osv-scanner: "osv.dev API error 429: too many requests" during a full-repo scan
Fixes osv-scanner getting rate-limited by the osv.dev API on large scans. Use it when a full-repo scan dies with 429s partway through. Key trigger: one giant scan firing off API queries for every lockfile at once.
TL;DR: Split the scan into smaller per-directory chunks and run them sequentially, letting each chunk's queries stay under the rate limit. A full-repo scan bursts hundreds of API calls at once and trips the limiter; smaller sequential scans spread the same work over time. Chunking is the fix.
osv.dev API error 429: too many requests- Confirm it is rate limiting and not an outage. Wait a minute and re-run a small single-lockfile scan.
Expected: the small scan succeeds. If even that 429s, the API itself is having a bad day and waiting is the only move.
- Split the repo scan into per-directory chunks, one lockfile or one package directory per invocation.
Expected: each chunk issues a fraction of the API calls of the full scan.
- Run the chunks sequentially, not in parallel, with a short pause between them if you still see 429s.
Expected: every chunk completes with a 200-level result instead of rate-limit errors.
- Combine the chunk outputs for your report or gate.
Expected: the union of findings matches what the full scan would have reported, without the 429 failures.
Use this when
- a full-repo osv-scanner run fails with API 429 errors
- CI scans many lockfiles in one invocation
- the 429s appear partway through an otherwise healthy scan
Not for this skill when
- every request 429s including tiny ones; that is an API incident or a blocked client, not scan size
- the error is a different API status (500s, timeouts); those are server or network problems
- you query the osv.dev API directly with your own client; tune your own client's retry logic instead
Variant phrasings
osv-scanner rate limited by osv.dev
too many requests during osv-scanner full repo scan
osv.dev API 429 on large scan
how to avoid osv-scanner rate limiting
Why it happens
osv-scanner turns every package it finds into queries against the osv.dev API, batched but still numerous. A full repo with many lockfiles produces a burst of requests that exceeds the API's per-client rate limit, and the API answers 429. The scanner cannot complete the run on partial answers, so the whole scan fails instead of returning incomplete results.
Edge cases
- Parallel CI jobs from the same egress IP share the limit; two repos scanning at once can 429 each other, so stagger scheduled scans.
- Newer osv-scanner releases retry transient 429s with backoff; if yours does not, upgrading is cheaper than building your own retry wrapper.
- Scanning one consolidated SBOM instead of a dozen lockfiles issues fewer queries for the same coverage; consider generating the SBOM once and scanning that.
- A persistent 429 across hours from a single small scan means something else is hammering the API from your network; check for a runaway loop before blaming the scanner.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst__M1Ef5DZpK6yaW58e-baGA