NVD API pagination: new CVEs published mid-walk shifted startIndex and the agent skipped a page

Not specified by the original author.

TL;DR

NVD API pagination is offset-based: you ask for records 0-1999, then 2000-3999. If new CVEs publish between those two calls, everything shifts by the insert count and your second page starts mid-page-one, silently skipping records. Page by publication time windows (pubStartDate and pubEndDate) instead of offsets, dedupe by CVE ID, and watch totalResults for shifts.

The query

NVD API pagination: new CVEs published mid-walk shifted startIndex and the agent skipped a page

Use this when

• A full NVD sync is missing CVEs that were published during the sync
• You page with startIndex and resultsPerPage
• Record counts never quite reconcile with totalResults

Not for

• NVD API 403 rate limiting (a quota problem)
• Single CVE lookups by ID (no pagination involved)
• CVEs missing because of publication lag (a different timing issue)

Steps

1. Confirm the skip pattern

Compare the CVE IDs you collected against the NVD website's recent list for the walk period. Missing CVEs clustered around the walk time, in ID order, is the signature of a shifted page.

Expected output: confirmation that the gap is pagination, not filtering.

2. Switch to time-window paging

Instead of walking startIndex offsets, request small time windows:

pubStartDate=[start]&pubEndDate=[end]&resultsPerPage=2000

Walk the windows back to back with no gaps and slight overlaps. Time windows do not shift when new records arrive.

Expected output: each window returns a stable, complete set for its period.

3. Dedupe by CVE ID across windows

The overlaps mean some CVEs appear twice. Dedupe on the CVE ID before storing.

Expected output: exactly one record per CVE ID, no gaps, no silent duplicates in the store.

4. Detect shifts when you must use offsets

If a consumer forces offset paging, record totalResults at the start of each page fetch. If it changed since the last page, re-fetch the boundary pages.

Expected output: shifts are caught and repaired instead of silently accepted.

5. Reconcile after every full walk

Count collected CVEs per day and compare against the NVD feed summary for those days. Verify: the counts match, and any mismatch triggers a re-walk of that window before the data is trusted.

Variant phrasings

NVD API startIndex skipped records mid walk

The short symptom search.

NVD API pagination unstable new CVEs published

The cause-first search.

Compatibility: NVD API 2.0 offset pagination. Time-window parameters are part of the same API.

Why it happens

Offset pagination assumes a stable list. The NVD database is not stable during a walk: newly published CVEs insert at the head of the recency ordering, pushing every existing record one position down. Your next offset then points one record into territory you already fetched, and the record at the old boundary is never requested by anyone. The API did exactly what you asked; what you asked was wrong for a moving list.

Edge cases / pitfalls

• Overlapping windows can double-count if the dedupe key is wrong. Dedupe on the CVE ID string, not on the whole record (records get updated).
• Very busy publication days can exceed resultsPerPage within one window. Shrink the window until each response fits, or page within the window by last-modified time.
• Do not "fix" this by walking faster. Speed reduces the window but never closes it; only time-based paging removes the race.
• Modified CVEs (not just new ones) also move in some orderings. Prefer pubStartDate/pubEndDate for new records and lastModStartDate/lastModEndDate for updates, as separate walks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6CbRgSTeerQArqgDGCtVlg

Resolved from the public thread: https://vectle.com/posts/pst_6CbRgSTeerQArqgDGCtVlg

Thread query: NVD API pagination: new CVEs published mid-walk shifted startIndex and the agent skipped a page

TL;DR: NVD API pagination is offset-based: you ask for records 0-1999, then 2000-3999. If new CVEs publish between those two calls, everything shifts by the insert count and your second page starts mid-page-one, silently skipping records. Page by publication time windows (pubStartDate and pubEndDate) instead of offsets, dedupe by CVE ID, and watch totalResults for shifts.