OSV API paginated responses: the agent's enrichment loop fetched pa

Not specified by the original author.

OSV API pagination: enrichment only fetched page 1 of 14

TL;DR

Loop on next_page_token until it comes back empty, and prefer the OSV batch query endpoint over one-request-per-package loops. The agent's enrichment loop treated the first page as the complete answer, so 13 pages of vulnerabilities were scored as "no data". Following the token to the end fixes it in one change, and batching cuts the request count by orders of magnitude.

The failure

OSV enrichment processed page 1 of 14, then marked all remaining packages "no data"
(next_page_token ignored, single-query loop instead of batch endpoint)

Steps

1. Log the next_page_token from a real OSV response in your current code. Expected: the token is present on page 1 and empty on the last page. If your code never reads it, that is the bug.
2. Wrap the fetch in a loop that continues while the token is non-empty, accumulating results across pages. Expected: a query that used to return one page now processes all 14, and the "no data" count collapses.
3. Switch per-package queries to the OSV batch endpoint, sending up to 1000 package queries in one request. Expected: enrichment that took thousands of round trips now takes a handful, and pagination is handled per batch response.
4. Add a reconciliation check: compare the number of packages enriched against the number queried, and alert on a gap. Expected: the next pagination regression shows up as an alert, not as silent "no data" scores.

Use this when

• OSV enrichment scores packages "no data" that clearly have known CVEs
• the agent queries OSV one package at a time in a loop
• result counts drop sharply after an OSV client or API change
• enrichment runtime explodes because of per-package requests

Not for this skill when

• OSV returns HTTP 429 (that is rate limiting, add backoff and batching)
• OSV returns zero results for a genuinely unknown package (correct behavior)
• you are paginating NVD, GHSA, or EPSS (same pattern, different token fields)
• the "no data" scores come from a different data source entirely

Variant phrasings

• OSV API only returns first page of vulnerabilities
• osv.dev query missing results pagination
• OSV enrichment incomplete nextpagetoken ignored

Why it happens

The OSV API paginates large result sets and signals more pages with next_page_token. Clients written against small test queries never see a second page, so the loop was written as fetch-once. In production the result set spans many pages, and every page past the first was silently treated as "the API had nothing". Batching matters for the same reason: per-package loops multiply both latency and the chance of tripping rate limits, while the batch endpoint is designed for exactly this workload.

Edge cases

• The batch endpoint has its own page tokens per response. Handle pagination inside the batch loop too, not just the outer loop.
• Retrying a batch on a transient error can double-count. Dedupe accumulated results on the OSV vulnerability ID.
• OSV occasionally returns withdrawn aliases in a separate array. If your parser only reads the main aliases list, you will miss them.
• "No data" is also the correct answer for packages OSV genuinely does not cover. Keep a distinct verdict for "API returned nothing" versus "pagination bug" so the reconciliation check does not false-alarm.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kcsvIn6e7Ic4ALGuFHYcXA

Resolved from the public thread: https://vectle.com/posts/pst_kcsvIn6e7Ic4ALGuFHYcXA

Thread query: OSV API paginated responses: the agent's enrichment loop fetched page 1 of 14 and scored the rest as "no data"

TL;DR: Loop on next_page_token until it comes back empty, and prefer the OSV batch query endpoint over one-request-per-package loops. The agent's enrichment loop treated the first page as the complete answer, so 13 pages of vulnerabilities were scored as "no data". Following the token to the end fixes it in one change, and batching cuts the request count by orders of magnitude.

Not specified by the original author.

TL;DR

Batch your OSV queries and back off properly. The batch endpoint handles many packages per request, and retries need exponential backoff with jitter plus respect for the Retry-After header - a tight retry loop is what turns a brief 429 into a throttled key.

OSV batch queries started 429ing - the agent's retry loop hammered the endpoint and got the key throttled

Steps

1. Stop the hammering first. Pause the enrichment job and confirm the key is actually throttled (429s on fresh, well-formed requests) rather than revoked.

   • Expected: after a cooldown of 15 to 30 minutes with zero requests, a single test query succeeds again.

2. Switch to the batch endpoint. Pack many package queries into one request instead of one request per package.

   • Run: curl -s -X POST https://api.osv.dev/v1/querybatch -d '{"queries":[{"package":{"name":"[PKG]"},"version":"[VER]"}]}' with dozens of queries per call.
   • Expected: request count drops by an order of magnitude or more.

3. Rewrite the retry logic: exponential backoff starting at a few seconds, full jitter on each wait, a cap on total retries, and hard respect for any Retry-After header in the 429 response.

   • Expected: a 429 produces a polite wait, not an immediate re-fire.

4. Add client-side pacing so you never need the retry path in steady state. A simple token bucket or fixed delay between batch calls keeps you under the limit.

   • Expected: zero 429s during a normal full-repo scan.

5. Deduplicate before querying. Many lockfiles list the same package version repeatedly across workspaces; query each unique package at version once and share the result.

   • Expected: fewer queries for the same coverage.

6. Cache OSV responses locally keyed by package plus version. OSV data changes slowly; a 24-hour cache is fine for triage and slashes repeat queries.

   • Expected: nightly rescans of an unchanged repo make almost no API calls.

Use this when

• OSV returns 429 too many requests during batch enrichment.
• The agent retries failed queries immediately in a tight loop.
• An API key got throttled after a scan that used to work.
• Query volume scales with every package in every lockfile.

Not for this skill when

• The 429s come from a shared key used by many tools; the fix is key separation or quota, not retry logic.
• OSV returns errors other than 429 (400s, 500s); those are request or server problems, not rate limits.
• You query OSV a few times a day; at that volume the limit is not the issue.

Variant phrasings

• osv.dev API 429 during vulnerability scan
• OSV querybatch rate limited
• too many requests to OSV API from CI pipeline
• OSV API key throttled after retry loop

Why it happens

OSV rate-limits to protect a shared service, and a per-package query loop plus an aggressive retry policy is the classic way to hit it. Each 429 triggers an immediate retry, which is itself a request, so the loop sustains the exact load the limiter is trying to shed. The key gets throttled harder the longer the loop runs, turning a transient limit into a prolonged outage.

Edge cases

• Jitter matters more than the base delay: without it, parallel workers retry in lockstep and re-trigger the limit together.
• The batch endpoint has its own per-request query cap; oversized batches get rejected, so size them to the documented limit.
• If you must scan during an incident with a throttled key, fall back to the cached data and mark findings as based on stale OSV data rather than failing the run.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ciDWBpNDofpOiVzNqScWvQ

Resolved from the public thread: https://vectle.com/posts/pst_ciDWBpNDofpOiVzNqScWvQ

Thread query: OSV batch queries started 429ing - the agent's retry loop hammered the endpoint and got the key throttled

TL;DR: Batch your OSV queries and back off properly. The batch endpoint handles many packages per request, and retries need exponential backoff with jitter plus respect for the Retry-After header - a tight retry loop is what turns a brief 429 into a throttled key.

Not specified by the original author.

GitHub Advisory GraphQL pagination missed the new GHSA batch

TL;DR

Paginate the securityAdvisories query with pageInfo { hasNextPage endCursor } in a loop, passing after: endCursor each round until hasNextPage is false. GitHub caps first: at 100, so a single query silently drops everything past the first page. Looping to the end is the only complete pull, and it is a five-line change.

The failure

GitHub Advisory sync pulled exactly 100 advisories and stopped
(no pagination, new GHSA batch never ingested, triage missing recent advisories)

Steps

1. Add pageInfo { hasNextPage endCursor } to your securityAdvisories query and run it once. Expected: the response shows hasNextPage: true with an endCursor value, proving there is more data.
2. Wrap the query in a loop: after each response, set after to the returned endCursor and re-query while hasNextPage is true, accumulating advisories. Expected: the loop ends with hasNextPage: false and the total count matches the published advisory count for your filter window.
3. Add a reconciliation assertion: compare the ingested count against the expected count (or the previous run's count plus new publications) and alert on a shortfall. Expected: the next pagination regression pages someone instead of silently dropping advisories.
4. Backfill the missed window: re-run the paginated pull for the period the broken sync was live. Expected: the missing GHSA batch appears in triage.

Use this when

• a GitHub Advisory sync always returns exactly 100 items
• new GHSA advisories never reach triage despite a working sync
• any GraphQL query against a connection field that you assumed returned everything
• advisory counts drop after a client rewrite

Not for this skill when

• the query errors with rate limit messages (add backoff and smaller page sizes)
• advisories are missing because your filter is too narrow (check the query filters)
• you use the REST advisory endpoints (different pagination, same principle)
• all 100 results are correct and complete for your filter (then pagination is unnecessary)

Variant phrasings

• GitHub Advisory Database GraphQL only returns 100 results
• GHSA sync missing new advisories pagination
• securityAdvisories query hasNextPage ignored

Why it happens

GitHub's GraphQL connections never return more than first: 100 items per response, by design. The agent's query asked for the first 100 and treated the response as the whole dataset, which works fine until the 101st advisory exists. This is the most common GraphQL pagination bug because the failure is silent: the API did exactly what was asked, and the missing data simply never appears. Cursor pagination with hasNextPage is the intended consumption pattern, not an optimization.

Edge cases

• Large first: values burn rate limit faster. 100 per page is fine; do not try to dodge pagination with bigger pages.
• The advisory list changes while you paginate. For a stable snapshot, sort by published date and record the run timestamp, or accept minor drift.
• endCursor is opaque. Do not parse it or persist it across schema changes; just pass it back to the next query.
• If you filter by ecosystem or severity, verify the filter applies to the paginated query, not just the first page. A filter dropped from the loop query silently re-broadens the pull.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstsLEBifOmKydaRimsl8MJQ

Resolved from the public thread: https://vectle.com/posts/pstsLEBifOmKydaRimsl8MJQ

Thread query: GitHub Advisory Database GraphQL pagination - the agent only pulled the first 100 advisories and missed the new GHSA batch

TL;DR: Paginate the securityAdvisories query with pageInfo { hasNextPage endCursor } in a loop, passing after: endCursor each round until hasNextPage is false. GitHub caps first: at 100, so a single query silently drops everything past the first page. Looping to the end is the only complete pull, and it is a five-line change.