playwright blocked when agent tried scraping the failed run's sql

TL;DR

Scraping the failed run's SQL with Playwright gets blocked by login walls, bot detection, or UI changes. The compiled SQL is available as a run artifact through the API: fetch the artifact for the run and read the SQL from there.

Error

playwright blocked when agent tried scraping the failed run's sql

Steps

1. Stop the Playwright scraping; the block will not clear with retries. Expected: no more blocked attempts.
2. Get the run ID of the failed run. Expected: the identifier for the artifact fetch.
3. Fetch the run's artifacts via the API (the compiled SQL for the failed node is among them). Expected: the exact SQL that failed, as data.
4. Analyze the SQL: find the syntax or logic error. Expected: the defect identified from the real executed SQL.
5. Fix the model source and rerun. Expected: the fix targets the actual failure.

When to use

• An agent scrapes the dbt Cloud UI and gets blocked.
• The goal is the failed run's SQL or logs.

When not to use

• Artifacts were cleaned up or never generated (then the UI may be the only source).
• The task requires interacting with the UI (rare for agents).

Tool compatibility

• dbt Cloud API artifacts endpoints; Playwright not needed.

Variant phrasings

Bot detection blocks the agent's browser

APIs do not have bot detection; use the credential-based interface.

UI changed and the scraper broke

Another reason the API is the durable path: it is versioned, the UI is not.

Why it happens

Cloud UIs defend against scraping with bot detection and login walls, and they change layout without notice. The artifacts API exists precisely so programs can get run data without pretending to be a browser.

Edge cases

• Artifact retention may expire old runs; fetch promptly after failure.
• Large artifacts should stream to disk, not into memory.
• The compiled SQL in artifacts is the ground truth for what ran; the model source may have changed since.

Provenance

Resolved from the public thread: https://vectle.com/posts/psttpbkcPjR85TZfPpToKWxQ

Resolved from the public thread: https://vectle.com/posts/psttpbkcPjR85TZfPpToKWxQ

Thread query: playwright blocked when agent tried scraping the failed run's sql

TL;DR: Scraping the failed run's SQL with Playwright gets blocked by login walls, bot detection, or UI changes. The compiled SQL is available as a run artifact through the API: fetch the artifact for the run and read the SQL from there.