the agent's per-CVE NVD lookups took 6 hours - switching to the bulk API changed the whole pipeline

TL;DR

Mirror NVD locally and enrich from the mirror. The bulk CVE feeds plus the paginated API give you the whole dataset in a few dozen requests; a scheduled sync keeps it fresh and per-CVE lookups disappear from the pipeline entirely.

the agent's per-CVE NVD lookups took 6 hours - switching to the bulk API changed the whole pipeline

Steps

1. Time the current pipeline and count its NVD requests. Confirm the hours are network round trips, not local processing.

   • Expected: request count roughly equals CVE count, and wall time tracks it.

2. Build the local mirror. Pull the NVD 2.0 CVE endpoint with a large page size, walking startIndex until you cover totalResults, and store each record keyed by CVE ID with its last-modified date.

   • Run: curl -s -H "apiKey value [YOUR-NVD-KEY]" "https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=2000&startIndex=0", page through, and load into your store.
   • Expected: the full dataset lands locally in dozens of requests, not thousands.

3. Rewrite the enrichment step to query the local store. No code path in the agent should call the NVD API per CVE anymore.

   • Expected: enrichment of the whole backlog runs in minutes against the local database.

4. Add incremental sync. On each sync, request only records modified since the last sync using the lastModStartDate and lastModEndDate parameters, and upsert them.

   • Expected: each sync touches only changed records - usually a small fraction.

5. Schedule the sync every few hours. NVD updates continuously; a 2 to 4 hour cadence keeps triage data fresh without wasting requests.

   • Expected: steady-state API usage is a handful of requests per cycle.

6. Stamp every enriched finding with the mirror's sync timestamp so consumers know the data's age.

   • Expected: findings carry an as-of date for their NVD data.

Use this when

• Per-CVE NVD lookups take hours and dominate pipeline runtime.
• Enrichment runs on a schedule over a large backlog.
• The same CVEs get looked up repeatedly across runs.
• API rate limits throttle the per-CVE pattern.

Not for this skill when

• You need real-time NVD data for a handful of CVEs; direct API calls are fine at low volume.
• The pipeline is slow for other reasons (LLM calls, SBOM generation); profile before rebuilding the NVD path.
• You cannot run a local store; consider the bulk JSON feeds downloaded on demand instead.

Variant phrasings

• NVD enrichment too slow for large CVE backlog
• how to bulk download NVD CVE data
• caching NVD API responses for vulnerability pipeline
• NVD incremental sync with lastModStartDate

Why it happens

Each per-CVE lookup pays a full HTTPS round trip plus rate-limit pacing, so 10,000 CVEs means 10,000 sequential waits. NVD's API is built for bulk transfer with big pages and modification-date filtering; the per-CVE loop ignores both features and pays the maximum possible cost. A local mirror flips the cost: one bulk transfer, then every lookup is a local query.

Edge cases

• The initial sync is large; run it once carefully and validate record counts against totalResults before relying on it.
• Deleted or rejected CVE records need handling in the upsert logic, or the mirror keeps stale live records.
• If the sync job fails silently, the mirror goes stale while looking healthy; alert on sync age, not just sync success.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gsDdpwz7hVKx30TCeikYXw