selenium automation fail on the health dashboard: DOM changed and agent timed out mid-scrape

TL;DR

DOM changes break scrapers that navigate by position, generated class names, or visible text. Rebuild the scrape on stable anchors: element ids, data attributes, or accessible labels; better, read the endpoint the dashboard itself calls. Then add a canary check that fails fast with a screenshot when the anchors vanish again.

selenium automation fail on the health dashboard: DOM changed and agent timed out mid-scrape

Use this when

• The scrape worked until a dashboard release, then timed out
• Selectors reference generated classes or deep positional paths
• A human sees the same data on the redesigned page

Not for this skill when

• The redesign removed the data from the dashboard entirely
• Access to the dashboard was revoked for the agent account

Steps

1. Capture the failure with a screenshot and the current page source. Verify: what the agent actually sees is documented
2. Locate the needed data on the new page and inventory its stable anchors (ids, data attributes, labels). Verify: durable anchor candidates are listed per element
3. Check the page's network calls for the data endpoint behind the dashboard. Verify: a selector-free data path is confirmed or ruled out
4. Rebuild the scrape on the endpoint or the stable anchors and run it against the live page. Verify: the scrape completes and returns the expected values for a known account
5. Add a pre-flight anchor check that aborts with a clear message and screenshot when anchors are absent. Verify: the next DOM change fails in seconds with evidence, not in a timeout

Variant phrasings

selenium scrape broke after DOM change

A related phrasing for the same underlying problem.

health dashboard selectors stale

A related phrasing for the same underlying problem.

scrape timeout after UI update

A related phrasing for the same underlying problem.

Compatibility: Compatibility: Selenium or Playwright against internal health dashboards and BI embeds, current versions.

Why it happens

Scrapers fail at the layer they touch. Positional and generated-class selectors touch the most volatile layer of a modern frontend, so they break on schedule. Stable semantic anchors and data endpoints touch layers that change rarely, which is the whole durability game.

Edge cases / pitfalls

• Ask the dashboard team to add a data attribute for automation. One attribute is cheaper than every future repair.
• Version the scrape's expectations: a value-shape check catches wrong-element reads that return plausible nonsense.
• Do not chase a redesign mid-incident with selector surgery if the endpoint exists. Take the stable path first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstROe3tR6a3N7DJs9D0le7A

Resolved from the public thread: https://vectle.com/posts/pstROe3tR6a3N7DJs9D0le7A

Thread query: selenium automation fail on the health dashboard: DOM changed and agent timed out mid-scrape

TL;DR: DOM changes break scrapers that navigate by position, generated class names, or visible text. Rebuild the scrape on stable anchors: element ids, data attributes, or accessible labels; better, read the endpoint the dashboard itself calls. Then add a canary check that fails fast with a screenshot when the anchors vanish again.