OSV API paginated responses: the agent's enrichment loop fetched page 1 of 14 and scored the rest as "no data"

OSV API pagination: enrichment only fetched page 1 of 14

TL;DR

Loop on next_page_token until it comes back empty, and prefer the OSV batch query endpoint over one-request-per-package loops. The agent's enrichment loop treated the first page as the complete answer, so 13 pages of vulnerabilities were scored as "no data". Following the token to the end fixes it in one change, and batching cuts the request count by orders of magnitude.

The failure

OSV enrichment processed page 1 of 14, then marked all remaining packages "no data"
(next_page_token ignored, single-query loop instead of batch endpoint)

Steps

1. Log the next_page_token from a real OSV response in your current code. Expected: the token is present on page 1 and empty on the last page. If your code never reads it, that is the bug.

1. Wrap the fetch in a loop that continues while the token is non-empty, accumulating results across pages. Expected: a query that used to return one page now processes all 14, and the "no data" count collapses.

1. Switch per-package queries to the OSV batch endpoint, sending up to 1000 package queries in one request. Expected: enrichment that took thousands of round trips now takes a handful, and pagination is handled per batch response.

1. Add a reconciliation check: compare the number of packages enriched against the number queried, and alert on a gap. Expected: the next pagination regression shows up as an alert, not as silent "no data" scores.

Use this when

• OSV enrichment scores packages "no data" that clearly have known CVEs
• the agent queries OSV one package at a time in a loop
• result counts drop sharply after an OSV client or API change
• enrichment runtime explodes because of per-package requests

Not for this skill when

• OSV returns HTTP 429 (that is rate limiting, add backoff and batching)
• OSV returns zero results for a genuinely unknown package (correct behavior)
• you are paginating NVD, GHSA, or EPSS (same pattern, different token fields)
• the "no data" scores come from a different data source entirely

Variant phrasings

• OSV API only returns first page of vulnerabilities
• osv.dev query missing results pagination
• OSV enrichment incomplete nextpagetoken ignored

Why it happens

The OSV API paginates large result sets and signals more pages with next_page_token. Clients written against small test queries never see a second page, so the loop was written as fetch-once. In production the result set spans many pages, and every page past the first was silently treated as "the API had nothing". Batching matters for the same reason: per-package loops multiply both latency and the chance of tripping rate limits, while the batch endpoint is designed for exactly this workload.

Edge cases

• The batch endpoint has its own page tokens per response. Handle pagination inside the batch loop too, not just the outer loop.
• Retrying a batch on a transient error can double-count. Dedupe accumulated results on the OSV vulnerability ID.
• OSV occasionally returns withdrawn aliases in a separate array. If your parser only reads the main aliases list, you will miss them.
• "No data" is also the correct answer for packages OSV genuinely does not cover. Keep a distinct verdict for "API returned nothing" versus "pagination bug" so the reconciliation check does not false-alarm.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kcsvIn6e7Ic4ALGuFHYcXA