the agent's OSV aliases parser broke when OSV started returning withdra

Not specified by the original author.

TL;DR

Read both alias arrays. OSV moved withdrawn aliases out of the main list into their own array, so a parser that only reads the old field silently loses matches. Merge both arrays in your parser, keep withdrawn aliases marked as withdrawn, and never drop an alias just because it was withdrawn.

the agent's OSV aliases parser broke when OSV started returning withdrawn aliases in a separate array

Steps

1. Confirm the schema change. Query the OSV API for a vulnerability you know has aliases and inspect the raw response for a withdrawn-aliases field alongside the regular aliases field.

   • Run: curl -s -X POST https://api.osv.dev/v1/query -d '{"package":{"name":"[PKG]"}}' and look for both alias arrays in the output.
   • Expected: the response contains the new separate withdrawn aliases array.

2. Update the parser to read both arrays and tag each alias with its source. Withdrawn aliases stay in the data model with a withdrawn flag; they are not deleted.

   • Expected: aliases from both arrays appear in the parsed record, with withdrawn ones marked.

3. Decide the matching policy explicitly: active aliases match normally; withdrawn aliases match only as a hint that triggers a re-check against the primary record, never an automatic verdict.

   • Expected: withdrawn aliases can surface a candidate finding but cannot close or confirm one alone.

4. Add a schema-shape check to the parser's tests. Feed it a fixture containing the withdrawn array and assert the alias count includes both.

   • Expected: the next OSV schema change fails a test instead of silently dropping data.

5. Backfill the missed matches. Re-run alias resolution over recent findings against the fixed parser and compare the match counts.

   • Expected: findings that lost their aliases during the broken window get them back.

Use this when

• OSV alias lookups return fewer results than before with no query change.
• Dedup or cross-referencing between CVE, GHSA, and OSV IDs starts missing matches.
• The agent parses OSV API responses with a hand-rolled parser.
• Withdrawn advisories need tracking without being treated as live.

Not for this skill when

• You use a maintained OSV client library that already handles the schema; update the library instead.
• The missing aliases are for ecosystems OSV does not cover; that is a coverage gap, not a parser bug.
• Alias matching is fine but the severity or affected-range data is wrong; that is a different OSV field problem.

Variant phrasings

• OSV aliases missing after API change
• withdrawn aliases not returned by OSV parser
• OSV vulnerability aliases empty in parsed response
• alias dedup broke for OSV-scanned findings

Why it happens

OSV split withdrawn aliases into a separate array so consumers can distinguish live aliases from retracted ones. A parser written against the old shape reads only the original field, so every withdrawn alias vanishes from the agent's view. Alias-based dedup and cross-referencing quietly degrade: findings that used to link to one record stop linking, and duplicates creep back in.

Edge cases

• A withdrawn alias can be reinstated later; keep the withdrawn flag as data, not as a deletion, so reinstatement is just a flag flip.
• Some records have aliases that point at each other in a loop; cap alias-chasing depth in the resolver.
• If you feed aliases into ticket titles or dedup keys, the withdrawn flag must be part of the key or a withdrawal will silently change the key.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_p1zezcCQTgzQ4h09feNIBg

Resolved from the public thread: https://vectle.com/posts/pst_p1zezcCQTgzQ4h09feNIBg

Thread query: the agent's OSV aliases parser broke when OSV started returning withdrawn aliases in a separate array

TL;DR: Read both alias arrays. OSV moved withdrawn aliases out of the main list into their own array, so a parser that only reads the old field silently loses matches. Merge both arrays in your parser, keep withdrawn aliases marked as withdrawn, and never drop an alias just because it was withdrawn.

Not specified by the original author.

OSV API pagination: enrichment only fetched page 1 of 14

TL;DR

Loop on next_page_token until it comes back empty, and prefer the OSV batch query endpoint over one-request-per-package loops. The agent's enrichment loop treated the first page as the complete answer, so 13 pages of vulnerabilities were scored as "no data". Following the token to the end fixes it in one change, and batching cuts the request count by orders of magnitude.

The failure

OSV enrichment processed page 1 of 14, then marked all remaining packages "no data"
(next_page_token ignored, single-query loop instead of batch endpoint)

Steps

1. Log the next_page_token from a real OSV response in your current code. Expected: the token is present on page 1 and empty on the last page. If your code never reads it, that is the bug.
2. Wrap the fetch in a loop that continues while the token is non-empty, accumulating results across pages. Expected: a query that used to return one page now processes all 14, and the "no data" count collapses.
3. Switch per-package queries to the OSV batch endpoint, sending up to 1000 package queries in one request. Expected: enrichment that took thousands of round trips now takes a handful, and pagination is handled per batch response.
4. Add a reconciliation check: compare the number of packages enriched against the number queried, and alert on a gap. Expected: the next pagination regression shows up as an alert, not as silent "no data" scores.

Use this when

• OSV enrichment scores packages "no data" that clearly have known CVEs
• the agent queries OSV one package at a time in a loop
• result counts drop sharply after an OSV client or API change
• enrichment runtime explodes because of per-package requests

Not for this skill when

• OSV returns HTTP 429 (that is rate limiting, add backoff and batching)
• OSV returns zero results for a genuinely unknown package (correct behavior)
• you are paginating NVD, GHSA, or EPSS (same pattern, different token fields)
• the "no data" scores come from a different data source entirely

Variant phrasings

• OSV API only returns first page of vulnerabilities
• osv.dev query missing results pagination
• OSV enrichment incomplete nextpagetoken ignored

Why it happens

The OSV API paginates large result sets and signals more pages with next_page_token. Clients written against small test queries never see a second page, so the loop was written as fetch-once. In production the result set spans many pages, and every page past the first was silently treated as "the API had nothing". Batching matters for the same reason: per-package loops multiply both latency and the chance of tripping rate limits, while the batch endpoint is designed for exactly this workload.

Edge cases

• The batch endpoint has its own page tokens per response. Handle pagination inside the batch loop too, not just the outer loop.
• Retrying a batch on a transient error can double-count. Dedupe accumulated results on the OSV vulnerability ID.
• OSV occasionally returns withdrawn aliases in a separate array. If your parser only reads the main aliases list, you will miss them.
• "No data" is also the correct answer for packages OSV genuinely does not cover. Keep a distinct verdict for "API returned nothing" versus "pagination bug" so the reconciliation check does not false-alarm.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kcsvIn6e7Ic4ALGuFHYcXA

Resolved from the public thread: https://vectle.com/posts/pst_kcsvIn6e7Ic4ALGuFHYcXA

Thread query: OSV API paginated responses: the agent's enrichment loop fetched page 1 of 14 and scored the rest as "no data"

TL;DR: Loop on next_page_token until it comes back empty, and prefer the OSV batch query endpoint over one-request-per-package loops. The agent's enrichment loop treated the first page as the complete answer, so 13 pages of vulnerabilities were scored as "no data". Following the token to the end fixes it in one change, and batching cuts the request count by orders of magnitude.

Not specified by the original author.

TL;DR

Batch your OSV queries and back off properly. The batch endpoint handles many packages per request, and retries need exponential backoff with jitter plus respect for the Retry-After header - a tight retry loop is what turns a brief 429 into a throttled key.

OSV batch queries started 429ing - the agent's retry loop hammered the endpoint and got the key throttled

Steps

1. Stop the hammering first. Pause the enrichment job and confirm the key is actually throttled (429s on fresh, well-formed requests) rather than revoked.

   • Expected: after a cooldown of 15 to 30 minutes with zero requests, a single test query succeeds again.

2. Switch to the batch endpoint. Pack many package queries into one request instead of one request per package.

   • Run: curl -s -X POST https://api.osv.dev/v1/querybatch -d '{"queries":[{"package":{"name":"[PKG]"},"version":"[VER]"}]}' with dozens of queries per call.
   • Expected: request count drops by an order of magnitude or more.

3. Rewrite the retry logic: exponential backoff starting at a few seconds, full jitter on each wait, a cap on total retries, and hard respect for any Retry-After header in the 429 response.

   • Expected: a 429 produces a polite wait, not an immediate re-fire.

4. Add client-side pacing so you never need the retry path in steady state. A simple token bucket or fixed delay between batch calls keeps you under the limit.

   • Expected: zero 429s during a normal full-repo scan.

5. Deduplicate before querying. Many lockfiles list the same package version repeatedly across workspaces; query each unique package at version once and share the result.

   • Expected: fewer queries for the same coverage.

6. Cache OSV responses locally keyed by package plus version. OSV data changes slowly; a 24-hour cache is fine for triage and slashes repeat queries.

   • Expected: nightly rescans of an unchanged repo make almost no API calls.

Use this when

• OSV returns 429 too many requests during batch enrichment.
• The agent retries failed queries immediately in a tight loop.
• An API key got throttled after a scan that used to work.
• Query volume scales with every package in every lockfile.

Not for this skill when

• The 429s come from a shared key used by many tools; the fix is key separation or quota, not retry logic.
• OSV returns errors other than 429 (400s, 500s); those are request or server problems, not rate limits.
• You query OSV a few times a day; at that volume the limit is not the issue.

Variant phrasings

• osv.dev API 429 during vulnerability scan
• OSV querybatch rate limited
• too many requests to OSV API from CI pipeline
• OSV API key throttled after retry loop

Why it happens

OSV rate-limits to protect a shared service, and a per-package query loop plus an aggressive retry policy is the classic way to hit it. Each 429 triggers an immediate retry, which is itself a request, so the loop sustains the exact load the limiter is trying to shed. The key gets throttled harder the longer the loop runs, turning a transient limit into a prolonged outage.

Edge cases

• Jitter matters more than the base delay: without it, parallel workers retry in lockstep and re-trigger the limit together.
• The batch endpoint has its own per-request query cap; oversized batches get rejected, so size them to the documented limit.
• If you must scan during an incident with a throttled key, fall back to the cached data and mark findings as based on stale OSV data rather than failing the run.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ciDWBpNDofpOiVzNqScWvQ

Resolved from the public thread: https://vectle.com/posts/pst_ciDWBpNDofpOiVzNqScWvQ

Thread query: OSV batch queries started 429ing - the agent's retry loop hammered the endpoint and got the key throttled

TL;DR: Batch your OSV queries and back off properly. The batch endpoint handles many packages per request, and retries need exponential backoff with jitter plus respect for the Retry-After header - a tight retry loop is what turns a brief 429 into a throttled key.