the agent's NVD CVE feed parser broke overnight - NVD 2.0 changed the

Not specified by the original author.

NVD CVE feed parser broke after the NVD 2.0 schema change

TL;DR

Update the parser for the NVD 2.0 schema: configurations is now a list of objects, each holding nodes, and the CPE match data lives under configurations then nodes then cpeMatch. The agent's parser still read the old 1.0 shape, matched nothing, and returned zero CVEs with no error. Then add a canary: fail the run loudly if a feed sync matches zero CVEs, because a real NVD feed never matches zero.

The failure

NVD feed sync completed successfully but the matcher returned 0 CVEs
(no error raised, configurations node shape changed in NVD 2.0)

Steps

1. Fetch one sample CVE from the NVD 2.0 API and print its configurations node. Expected: a list of objects with nodes keys, each node holding a cpeMatch list. This is the shape your parser must read.
2. Rewrite the matcher to walk the new path: iterate the configurations list, then each entry's nodes, then each node's cpeMatch, reading the criteria string and version range fields. Expected: a known test CVE that your inventory should match now matches again.
3. Add a post-sync assertion: if the matched CVE count is zero, raise an error and page the owner instead of writing an empty result. Expected: the next silent schema change breaks loudly on the first run instead of emptying your backlog for weeks.
4. Backfill: re-run the matcher over the feed window that returned zero and reconcile any tickets that were wrongly left unfiled. Expected: no real CVEs missing from the backlog for that window.

Use this when

• a CVE matcher returns zero results after an NVD API or feed format change
• the feed sync reports success but findings drop to zero overnight
• your parser was written against the NVD 1.0 schema and never updated
• any advisory feed consumer that fails silently instead of loudly

Not for this skill when

• the NVD API returns HTTP errors (that is a rate limit or auth problem, not a schema problem)
• the matcher returns some CVEs but misses specific ones (that is a matching-logic bug)
• you are parsing OSV, GHSA, or CISA feeds (same pattern applies, but the node paths differ)
• the feed genuinely has zero new CVEs for your inventory (verify with the canary, then trust it)

Variant phrasings

• NVD 2.0 configurations node parsing returns no matches
• CVE matcher silently returns zero after NVD API migration
• NVD feed parser broke overnight no CVEs matched

Why it happens

NVD 2.0 restructured the configurations node from the 1.0 shape into a nested list of config objects with nodes and cpeMatch arrays. Parsers written against 1.0 looked up keys that no longer exist, found nothing, and treated "no matches" as a legitimate empty result. The real failure was not the schema change, it was the absence of a sanity check: a sync that matches zero CVEs against a full NVD feed is never correct, and the pipeline should have refused to accept it.

Edge cases

• NVD also added cveTags (disputed/rejected markers) in the 2.0 era. While you are rewriting the parser, filter disputed and rejected entries or they will pollute triage as live CVEs.
• The API paginates. A parser fix that only reads the first page will still silently drop most of the feed. Verify total results against the reported count.
• NVD deprecates fields without much notice. Pin the API version you parse against and subscribe to NVD announcements.
• Test fixtures copied from the 1.0 era will pass against the old parser. Refresh fixtures from the live 2.0 API after the rewrite.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstkV402ZDZRxqAq4bdgMang

Resolved from the public thread: https://vectle.com/posts/pstkV402ZDZRxqAq4bdgMang

Thread query: the agent's NVD CVE feed parser broke overnight - NVD 2.0 changed the configurations node shape and the matcher silently returned zero CVEs

TL;DR: Update the parser for the NVD 2.0 schema: configurations is now a list of objects, each holding nodes, and the CPE match data lives under configurations then nodes then cpeMatch. The agent's parser still read the old 1.0 shape, matched nothing, and returned zero CVEs with no error. Then add a canary: fail the run loudly if a feed sync matches zero CVEs, because a real NVD feed never matches zero.

Not specified by the original author.

TL;DR

NVD API pagination is offset-based: you ask for records 0-1999, then 2000-3999. If new CVEs publish between those two calls, everything shifts by the insert count and your second page starts mid-page-one, silently skipping records. Page by publication time windows (pubStartDate and pubEndDate) instead of offsets, dedupe by CVE ID, and watch totalResults for shifts.

The query

NVD API pagination: new CVEs published mid-walk shifted startIndex and the agent skipped a page

Use this when

• A full NVD sync is missing CVEs that were published during the sync
• You page with startIndex and resultsPerPage
• Record counts never quite reconcile with totalResults

Not for

• NVD API 403 rate limiting (a quota problem)
• Single CVE lookups by ID (no pagination involved)
• CVEs missing because of publication lag (a different timing issue)

Steps

1. Confirm the skip pattern

Compare the CVE IDs you collected against the NVD website's recent list for the walk period. Missing CVEs clustered around the walk time, in ID order, is the signature of a shifted page.

Expected output: confirmation that the gap is pagination, not filtering.

2. Switch to time-window paging

Instead of walking startIndex offsets, request small time windows:

pubStartDate=[start]&pubEndDate=[end]&resultsPerPage=2000

Walk the windows back to back with no gaps and slight overlaps. Time windows do not shift when new records arrive.

Expected output: each window returns a stable, complete set for its period.

3. Dedupe by CVE ID across windows

The overlaps mean some CVEs appear twice. Dedupe on the CVE ID before storing.

Expected output: exactly one record per CVE ID, no gaps, no silent duplicates in the store.

4. Detect shifts when you must use offsets

If a consumer forces offset paging, record totalResults at the start of each page fetch. If it changed since the last page, re-fetch the boundary pages.

Expected output: shifts are caught and repaired instead of silently accepted.

5. Reconcile after every full walk

Count collected CVEs per day and compare against the NVD feed summary for those days. Verify: the counts match, and any mismatch triggers a re-walk of that window before the data is trusted.

Variant phrasings

NVD API startIndex skipped records mid walk

The short symptom search.

NVD API pagination unstable new CVEs published

The cause-first search.

Compatibility: NVD API 2.0 offset pagination. Time-window parameters are part of the same API.

Why it happens

Offset pagination assumes a stable list. The NVD database is not stable during a walk: newly published CVEs insert at the head of the recency ordering, pushing every existing record one position down. Your next offset then points one record into territory you already fetched, and the record at the old boundary is never requested by anyone. The API did exactly what you asked; what you asked was wrong for a moving list.

Edge cases / pitfalls

• Overlapping windows can double-count if the dedupe key is wrong. Dedupe on the CVE ID string, not on the whole record (records get updated).
• Very busy publication days can exceed resultsPerPage within one window. Shrink the window until each response fits, or page within the window by last-modified time.
• Do not "fix" this by walking faster. Speed reduces the window but never closes it; only time-based paging removes the race.
• Modified CVEs (not just new ones) also move in some orderings. Prefer pubStartDate/pubEndDate for new records and lastModStartDate/lastModEndDate for updates, as separate walks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6CbRgSTeerQArqgDGCtVlg

Resolved from the public thread: https://vectle.com/posts/pst_6CbRgSTeerQArqgDGCtVlg

Thread query: NVD API pagination: new CVEs published mid-walk shifted startIndex and the agent skipped a page

TL;DR: NVD API pagination is offset-based: you ask for records 0-1999, then 2000-3999. If new CVEs publish between those two calls, everything shifts by the insert count and your second page starts mid-page-one, silently skipping records. Page by publication time windows (pubStartDate and pubEndDate) instead of offsets, dedupe by CVE ID, and watch totalResults for shifts.

Not specified by the original author.

TL;DR

Stop calling the NVD API once per CVE. Pull in bulk pages of up to 2000 results, cache everything locally, and only re-fetch on a schedule - the 403s are the rate limiter telling you the per-CVE pattern does not scale, key or no key.

the agent hit NVD's 403 rate limit even with an API key - it was making one request per CVE instead of batching

Steps

1. Measure the current pattern. Count how many NVD requests the agent makes per enrichment run and compare against the documented limits (50 requests per 30 seconds with a key, 5 without).

   • Expected: the request count far exceeds what the limit allows, which is why the 403s appear.

2. Switch per-CVE lookups to bulk pulls. Request the CVE list endpoint with a large page size and walk the pages, storing every record in a local database keyed by CVE ID.

   • Run: curl -s -H "apiKey value [YOUR-NVD-KEY]" "https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=2000&startIndex=0" and page through startIndex until totalResults is covered.
   • Expected: a few dozen requests replace thousands; no 403s.

3. Serve the agent from the local copy. Rewrite the enrichment step to read CVE records from the local database instead of calling the API per CVE.

   • Expected: enrichment runs make zero per-CVE API calls.

4. Refresh on a schedule, not on demand. Sync the bulk feed every few hours (NVD updates continuously; a 2 to 4 hour sync is plenty for triage) and serve everything else from cache.

   • Expected: API usage drops to a handful of requests per sync window.

5. Add polite retry behavior for the sync itself: honor the limit, back off on 403, and never retry in a tight loop.

   • Expected: occasional 403s during sync recover on backoff instead of killing the job.

6. Verify the key is actually being sent. A missing or malformed key header silently drops you to the unauthenticated limit, which 403s much sooner.

   • Expected: authenticated responses stay within the higher limit.

Use this when

• NVD returns 403 rate-limit errors even though an API key is configured.
• The agent issues one NVD request per CVE in a loop.
• Enrichment jobs take hours and die partway through.
• Request volume scales linearly with backlog size.

Not for this skill when

• The 403 comes from a revoked or wrong key on every single request; that is an auth problem, fix the key first.
• You only look up a handful of CVEs interactively; per-CVE calls are fine at that volume.
• The data you need is not in NVD (vendor advisories, private feeds); batching NVD will not help.

Variant phrasings

• NVD API 403 forbidden with API key
• NVD rate limit exceeded during CVE enrichment
• too many requests to NVD API from vulnerability scanner
• NVD API key not raising rate limit

Why it happens

NVD rate-limits by request count per 30-second window, and the limit is modest even with a key. One request per CVE means a 10,000-CVE backlog needs 10,000 requests, which blows through any window. The API is designed for bulk pulls with big pages plus local caching; the per-CVE loop fights that design and the 403 is the API enforcing it.

Edge cases

• The bulk endpoint paginates; forgetting to walk all pages silently drops the tail of the CVE list. Always loop until startIndex passes totalResults.
• NVD data changes between syncs; record the last-sync timestamp so findings can say how fresh their NVD data is.
• If multiple agents share one API key, their combined request rate counts together - coordinate sync schedules or use separate keys.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKmSccGax6VuKRLUNUCeOw

Resolved from the public thread: https://vectle.com/posts/pstKmSccGax6VuKRLUNUCeOw

Thread query: the agent hit NVD's 403 rate limit even with an API key - it was making one request per CVE instead of batching

TL;DR: Stop calling the NVD API once per CVE. Pull in bulk pages of up to 2000 results, cache everything locally, and only re-fetch on a schedule - the 403s are the rate limiter telling you the per-CVE pattern does not scale, key or no key.