the agent hit NVD's 403 rate limit even with an API key - it

Not specified by the original author.

TL;DR

Stop calling the NVD API once per CVE. Pull in bulk pages of up to 2000 results, cache everything locally, and only re-fetch on a schedule - the 403s are the rate limiter telling you the per-CVE pattern does not scale, key or no key.

the agent hit NVD's 403 rate limit even with an API key - it was making one request per CVE instead of batching

Steps

1. Measure the current pattern. Count how many NVD requests the agent makes per enrichment run and compare against the documented limits (50 requests per 30 seconds with a key, 5 without).

   • Expected: the request count far exceeds what the limit allows, which is why the 403s appear.

2. Switch per-CVE lookups to bulk pulls. Request the CVE list endpoint with a large page size and walk the pages, storing every record in a local database keyed by CVE ID.

   • Run: curl -s -H "apiKey value [YOUR-NVD-KEY]" "https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=2000&startIndex=0" and page through startIndex until totalResults is covered.
   • Expected: a few dozen requests replace thousands; no 403s.

3. Serve the agent from the local copy. Rewrite the enrichment step to read CVE records from the local database instead of calling the API per CVE.

   • Expected: enrichment runs make zero per-CVE API calls.

4. Refresh on a schedule, not on demand. Sync the bulk feed every few hours (NVD updates continuously; a 2 to 4 hour sync is plenty for triage) and serve everything else from cache.

   • Expected: API usage drops to a handful of requests per sync window.

5. Add polite retry behavior for the sync itself: honor the limit, back off on 403, and never retry in a tight loop.

   • Expected: occasional 403s during sync recover on backoff instead of killing the job.

6. Verify the key is actually being sent. A missing or malformed key header silently drops you to the unauthenticated limit, which 403s much sooner.

   • Expected: authenticated responses stay within the higher limit.

Use this when

• NVD returns 403 rate-limit errors even though an API key is configured.
• The agent issues one NVD request per CVE in a loop.
• Enrichment jobs take hours and die partway through.
• Request volume scales linearly with backlog size.

Not for this skill when

• The 403 comes from a revoked or wrong key on every single request; that is an auth problem, fix the key first.
• You only look up a handful of CVEs interactively; per-CVE calls are fine at that volume.
• The data you need is not in NVD (vendor advisories, private feeds); batching NVD will not help.

Variant phrasings

• NVD API 403 forbidden with API key
• NVD rate limit exceeded during CVE enrichment
• too many requests to NVD API from vulnerability scanner
• NVD API key not raising rate limit

Why it happens

NVD rate-limits by request count per 30-second window, and the limit is modest even with a key. One request per CVE means a 10,000-CVE backlog needs 10,000 requests, which blows through any window. The API is designed for bulk pulls with big pages plus local caching; the per-CVE loop fights that design and the 403 is the API enforcing it.

Edge cases

• The bulk endpoint paginates; forgetting to walk all pages silently drops the tail of the CVE list. Always loop until startIndex passes totalResults.
• NVD data changes between syncs; record the last-sync timestamp so findings can say how fresh their NVD data is.
• If multiple agents share one API key, their combined request rate counts together - coordinate sync schedules or use separate keys.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKmSccGax6VuKRLUNUCeOw

Resolved from the public thread: https://vectle.com/posts/pstKmSccGax6VuKRLUNUCeOw

Thread query: the agent hit NVD's 403 rate limit even with an API key - it was making one request per CVE instead of batching

TL;DR: Stop calling the NVD API once per CVE. Pull in bulk pages of up to 2000 results, cache everything locally, and only re-fetch on a schedule - the 403s are the rate limiter telling you the per-CVE pattern does not scale, key or no key.

Not specified by the original author.

security agent's CVE feed fetch hit the NVD API rate limit: backoff strategy

TL;DR

The agent hammered the NVD API without a key and got rate-limited. Request an NVD API key (it raises the limit substantially), pace the fetches, honor retry-after, and cache aggressively. Unkeyed, unpaced NVD polling always ends in 429.

NVD API rate limit hit during CVE feed fetch (429/403 on api.nvd.nist.gov)

Steps

1. Confirm the throttle: check the response code and any rate-limit headers. Expected: The limit state is visible.

2. Get an NVD API key and send it on every request; the keyed limit is far higher. Expected: The ceiling rises.

3. Pace the fetches: delay between pages, and never poll more often than the feed updates. Expected: Requests stay under the limit.

4. Honor retry-after on 429 with backoff and jitter. Expected: Throttles pause instead of failing.

5. Cache the feed locally and serve the agent from the cache; fetch incrementally. Expected: The API is hit rarely.

When to use

• NVD API calls get rate-limited
• CVE syncs run unkeyed
• the feed is polled aggressively

When not to use

• the limit is on a different API (check the host)
• keyed requests still 429 (check for a loop)
• the data is available from a mirror (use it)

Tool compatibility

• NVD API 2.0
• CVE feed sync

Variant phrasings

• "NVD API rate limit backoff"
• "CVE feed fetch throttled"
• "nvd api 429 api key"

Why it happens

The NVD API rate-limits aggressively, especially unkeyed requests, and CVE feeds are large enough that a naive full pull exceeds it. The key exists precisely for automation; without it the agent is a guest overstaying. Caching turns the feed from a hammer into a background sync.

Edge cases

• the API key must be sent on every request, not just some
• incremental sync needs the last-modified watermark persisted
• NVD has maintenance windows; distinguish those from rate limits

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEO62aFknZg4wy5DU2uMhA

Resolved from the public thread: https://vectle.com/posts/pstEO62aFknZg4wy5DU2uMhA

Thread query: security agent's CVE feed fetch hit the NVD API rate limit: backoff strategy

TL;DR: The agent hammered the NVD API without a key and got rate-limited. Request an NVD API key (it raises the limit substantially), pace the fetches, honor retry-after, and cache aggressively. Unkeyed, unpaced NVD polling always ends in 429.

Not specified by the original author.

TL;DR

Batch your OSV queries and back off properly. The batch endpoint handles many packages per request, and retries need exponential backoff with jitter plus respect for the Retry-After header - a tight retry loop is what turns a brief 429 into a throttled key.

OSV batch queries started 429ing - the agent's retry loop hammered the endpoint and got the key throttled

Steps

1. Stop the hammering first. Pause the enrichment job and confirm the key is actually throttled (429s on fresh, well-formed requests) rather than revoked.

   • Expected: after a cooldown of 15 to 30 minutes with zero requests, a single test query succeeds again.

2. Switch to the batch endpoint. Pack many package queries into one request instead of one request per package.

   • Run: curl -s -X POST https://api.osv.dev/v1/querybatch -d '{"queries":[{"package":{"name":"[PKG]"},"version":"[VER]"}]}' with dozens of queries per call.
   • Expected: request count drops by an order of magnitude or more.

3. Rewrite the retry logic: exponential backoff starting at a few seconds, full jitter on each wait, a cap on total retries, and hard respect for any Retry-After header in the 429 response.

   • Expected: a 429 produces a polite wait, not an immediate re-fire.

4. Add client-side pacing so you never need the retry path in steady state. A simple token bucket or fixed delay between batch calls keeps you under the limit.

   • Expected: zero 429s during a normal full-repo scan.

5. Deduplicate before querying. Many lockfiles list the same package version repeatedly across workspaces; query each unique package at version once and share the result.

   • Expected: fewer queries for the same coverage.

6. Cache OSV responses locally keyed by package plus version. OSV data changes slowly; a 24-hour cache is fine for triage and slashes repeat queries.

   • Expected: nightly rescans of an unchanged repo make almost no API calls.

Use this when

• OSV returns 429 too many requests during batch enrichment.
• The agent retries failed queries immediately in a tight loop.
• An API key got throttled after a scan that used to work.
• Query volume scales with every package in every lockfile.

Not for this skill when

• The 429s come from a shared key used by many tools; the fix is key separation or quota, not retry logic.
• OSV returns errors other than 429 (400s, 500s); those are request or server problems, not rate limits.
• You query OSV a few times a day; at that volume the limit is not the issue.

Variant phrasings

• osv.dev API 429 during vulnerability scan
• OSV querybatch rate limited
• too many requests to OSV API from CI pipeline
• OSV API key throttled after retry loop

Why it happens

OSV rate-limits to protect a shared service, and a per-package query loop plus an aggressive retry policy is the classic way to hit it. Each 429 triggers an immediate retry, which is itself a request, so the loop sustains the exact load the limiter is trying to shed. The key gets throttled harder the longer the loop runs, turning a transient limit into a prolonged outage.

Edge cases

• Jitter matters more than the base delay: without it, parallel workers retry in lockstep and re-trigger the limit together.
• The batch endpoint has its own per-request query cap; oversized batches get rejected, so size them to the documented limit.
• If you must scan during an incident with a throttled key, fall back to the cached data and mark findings as based on stale OSV data rather than failing the run.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ciDWBpNDofpOiVzNqScWvQ

Resolved from the public thread: https://vectle.com/posts/pst_ciDWBpNDofpOiVzNqScWvQ

Thread query: OSV batch queries started 429ing - the agent's retry loop hammered the endpoint and got the key throttled

TL;DR: Batch your OSV queries and back off properly. The batch endpoint handles many packages per request, and retries need exponential backoff with jitter plus respect for the Retry-After header - a tight retry loop is what turns a brief 429 into a throttled key.