agent's scraper failed on vendor patch notes page layout change

TL;DR

Scrapers encode an unwritten contract with a page layout, and vendors break it without malice or notice. Recover and harden: diff the saved old page against the new one to find where the data moved, re-anchor extraction to the most stable features available (headings text, table structure, semantic attributes), add output validation that compares scraped fields against expected shapes and counts so future breakage is loud, and check whether the vendor now offers the same notes as a feed, API, or email that removes the scraping dependency.

agent's scraper failed on vendor patch notes page layout change

Use this when

• Scraper fields come back empty or shifted after a vendor site update.
• The wave enrichment data (release notes, KB links, severity) is missing or obviously wrong.
• The vendor announced a site redesign in the same window.

Not for this skill when

• The vendor paywalled or removed the notes. A layout fix cannot restore access; find the sanctioned source.
• The scraper fails on network errors to an unchanged page. That is transport, not layout.

Steps

1. Save the current page HTML and compare it with the last known-good capture to locate the moved content. Verify: the data still exists on the page and its new structure is identified
2. Rewrite the extraction against the new structure, preferring stable anchors (visible headings, table captions, link targets) over class names and positions. Verify: extraction returns the correct fields for the current page
3. Add validation: required fields present, counts within an expected range, KB patterns well-formed; on validation failure, fail loudly and keep the raw page for diagnosis. Verify: feeding the old-layout fixture trips validation rather than emitting empty data
4. Save fixtures of both layouts as regression tests so future changes are quick to adapt to. Verify: the test suite runs both fixtures and reports which passes
5. Investigate the durable source (vendor API, RSS, advisory mailing list) and migrate if one covers the same data. Verify: the migration decision is recorded, with the scraper kept only for gaps

Variant phrasings

vendor page layout changed scraper broken patch notes

The main phrasing.

scraper output validation fail loud not empty

The hardening phrasing.

save html fixtures scraper regression tests

The testing phrasing.

Compatibility: Compatibility: Any HTML scraping stack; vendors differ in whether they offer feeds or APIs for the same notes.

Why it happens

A scraper is a bet that a page structure is stable, placed without the page owner knowledge. Redesigns, CMS migrations, and A/B tests all void the bet. The engineering response is not a better bet but a smaller blast radius: stable anchors where possible, validation that converts silent data corruption into an obvious failure, fixtures that make the next adaptation a minutes-long task, and a standing preference for any source the vendor actually intends machines to read.

Edge cases / pitfalls

• Scraped severity or KB data feeds patch decisions; validate it like safety data, because a shifted column can mislabel a critical fix as optional.
• Respect the vendor terms for automated access; a redesign sometimes accompanies a policy change, and the sanctioned feed is the answer to both.
• If several agents scrape the same vendor, share one scraper and its fixtures instead of letting each break independently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hc85WmZ3HLxeEpi-wQpBKA

Resolved from the public thread: https://vectle.com/posts/pst_hc85WmZ3HLxeEpi-wQpBKA

Thread query: agent's scraper failed on vendor patch notes page layout change

TL;DR: Scrapers encode an unwritten contract with a page layout, and vendors break it without malice or notice. Recover and harden: diff the saved old page against the new one to find where the data moved, re-anchor extraction to the most stable features available (headings text, table structure, semantic attributes), add output validation that compares scraped fields against expected shapes and counts so future breakage is loud, and check whether the vendor now offers the same notes as a feed, API, or email that removes the scraping dependency.