the agent's Debian security tracker scrape broke when the tracker moved to the new JSON format

Debian security tracker scrape broke on the new JSON format

TL;DR

Stop scraping the tracker's HTML pages and switch to its JSON data endpoint, parsing the documented JSON keys instead of page markup. The tracker moved to a JSON-first format, which broke every hand-rolled HTML scraper at once. The JSON endpoint is a stable, versioned contract keyed by source package then CVE, while the HTML layout was never a contract at all.

The failure

Debian security tracker scrape returned empty results after the tracker format change
(HTML selectors matched nothing, parser silently produced zero findings)

Steps

1. Fetch the tracker's JSON data endpoint and pretty-print one package entry. Expected: a stable structure keyed by source package, with per-CVE status entries under each package.

1. Rewrite the parser against the JSON keys: look up the package, then iterate its CVE entries reading status and fixed-version fields. Expected: the same CVEs match as before the break, verified against a known-vulnerable package.

1. Add a load-time validation that checks the expected top-level keys exist and fails loudly if the shape is unrecognized. Expected: the next format change raises an alert instead of silently returning zero findings.

1. Backfill the window the scraper was broken: re-run the JSON parser over that period and reconcile missing findings. Expected: no Debian CVEs missing from the backlog for the broken window.

Use this when

• a Debian security tracker scrape returns empty after a tracker update
• HTML selectors that used to work suddenly match nothing
• the agent parses security-tracker.debian.org pages by hand
• any hand-scraped security feed breaks when the site redesigns

Not for this skill when

• the JSON endpoint is unreachable (check network and the endpoint URL first)
• you need data the JSON feed does not carry (then scrape selectively, and expect breakage)
• the breakage is in NVD, OSV, or GHSA parsing (different feeds, different fixes)
• results are wrong rather than empty (that is a parsing-logic bug, not a format change)

Variant phrasings

• security-tracker.debian.org scraper stopped working
• Debian tracker JSON format parser empty results
• Debian CVE tracker HTML scraping broke

Why it happens

The Debian security tracker moved to a JSON-first data format, and HTML scrapers depend on markup that changed with the move. Scraping was always fragile: every layout tweak, class rename, or template change silently breaks selectors. The JSON endpoint exists precisely so consumers do not have to scrape, and its keys change far less often than page markup. The deeper fix is the validation step. The scrape did not error when it broke, it just returned nothing, which is why the outage lasted until someone noticed the empty backlog.

Edge cases

• The JSON feed is large. Do not fetch and parse it on every triage run; cache it locally and refresh on a schedule.
• Package name mismatches (source package versus binary package) cause silent misses. Make sure your lookup key matches the feed's keying.
• The tracker marks some CVEs as not-affected or no-dsa. Carry those statuses through instead of dropping them, or you will re-triage settled items.
• If you still need fields only present in HTML (rare), isolate that scrape to the smallest possible surface and alert on selector misses.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstPwQdlJnD62FFgMeuGA4tg