the agent hit NVD's 403 rate limit even with an API key - it was making one request per CVE instead of batching

TL;DR

Stop calling the NVD API once per CVE. Pull in bulk pages of up to 2000 results, cache everything locally, and only re-fetch on a schedule - the 403s are the rate limiter telling you the per-CVE pattern does not scale, key or no key.

the agent hit NVD's 403 rate limit even with an API key - it was making one request per CVE instead of batching

Steps

1. Measure the current pattern. Count how many NVD requests the agent makes per enrichment run and compare against the documented limits (50 requests per 30 seconds with a key, 5 without).

• Expected: the request count far exceeds what the limit allows, which is why the 403s appear.

1. Switch per-CVE lookups to bulk pulls. Request the CVE list endpoint with a large page size and walk the pages, storing every record in a local database keyed by CVE ID.

• Run: curl -s -H "apiKey value [YOUR-NVD-KEY]" "https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=2000&startIndex=0" and page through startIndex until totalResults is covered.
• Expected: a few dozen requests replace thousands; no 403s.

1. Serve the agent from the local copy. Rewrite the enrichment step to read CVE records from the local database instead of calling the API per CVE.

• Expected: enrichment runs make zero per-CVE API calls.

1. Refresh on a schedule, not on demand. Sync the bulk feed every few hours (NVD updates continuously; a 2 to 4 hour sync is plenty for triage) and serve everything else from cache.

• Expected: API usage drops to a handful of requests per sync window.

1. Add polite retry behavior for the sync itself: honor the limit, back off on 403, and never retry in a tight loop.

• Expected: occasional 403s during sync recover on backoff instead of killing the job.

1. Verify the key is actually being sent. A missing or malformed key header silently drops you to the unauthenticated limit, which 403s much sooner.

• Expected: authenticated responses stay within the higher limit.

Use this when

• NVD returns 403 rate-limit errors even though an API key is configured.
• The agent issues one NVD request per CVE in a loop.
• Enrichment jobs take hours and die partway through.
• Request volume scales linearly with backlog size.

Not for this skill when

• The 403 comes from a revoked or wrong key on every single request; that is an auth problem, fix the key first.
• You only look up a handful of CVEs interactively; per-CVE calls are fine at that volume.
• The data you need is not in NVD (vendor advisories, private feeds); batching NVD will not help.

Variant phrasings

• NVD API 403 forbidden with API key
• NVD rate limit exceeded during CVE enrichment
• too many requests to NVD API from vulnerability scanner
• NVD API key not raising rate limit

Why it happens

NVD rate-limits by request count per 30-second window, and the limit is modest even with a key. One request per CVE means a 10,000-CVE backlog needs 10,000 requests, which blows through any window. The API is designed for bulk pulls with big pages plus local caching; the per-CVE loop fights that design and the 403 is the API enforcing it.

Edge cases

• The bulk endpoint paginates; forgetting to walk all pages silently drops the tail of the CVE list. Always loop until startIndex passes totalResults.
• NVD data changes between syncs; record the last-sync timestamp so findings can say how fresh their NVD data is.
• If multiple agents share one API key, their combined request rate counts together - coordinate sync schedules or use separate keys.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKmSccGax6VuKRLUNUCeOw