playwright TimeoutError waiting for selector during scrape

TL;DR

Playwright's default 30-second selector wait is strict: if the element is hidden, detached, or never renders, you get TimeoutError. Wait for the right state (visible vs attached), confirm the selector matches in the live DOM, and only then raise the timeout.

playwright TimeoutError waiting for selector during scrape

Use this when

• wait_for_selector or a locator action throws TimeoutError.
• The element shows in a manual browser but not in the script.
• Scrapes work on some pages and fail on others.

Not for this skill when

• Navigation itself times out. That is a navigation problem.
• The page returns an error status. Fix the request first.

Steps

1. Print what the page actually rendered: print(await page.content()) and search for your selector text. Verify: you can see whether the element exists at all.
2. Test the selector in the browser devtools console on the live site. Verify: document.querySelector finds it there.
3. Match the wait to the need: await page.wait_for_selector('.price', state='visible', timeout=10000). Verify: no timeout on a page where the element renders.
4. If the element renders late (lazy load, infinite scroll), scroll or wait for network idle first, then wait for the selector. Verify: the element appears after the scroll.
5. For flaky dynamic pages, wrap the wait in a retry loop of 3 attempts with a page reload between attempts. Verify: the scrape succeeds on at least 9 of 10 runs.

Variant phrasings

playwright waitforselector timeout

Same failure, API-name search.

locator timeout playwright scrape

The locator-API variant.

playwright element not found timeout

Symptom-level search.

Compatibility: Playwright for Python and Node. Timeouts are in milliseconds in both. Default is 30000.

Why it happens

Playwright waits until the selector matches AND the element reaches the requested state. Pages that lazy-load, A/B test layouts, or render behind auth show a different DOM to the script than to your manual browser, so the selector never matches and the wait burns its full timeout.

Edge cases / pitfalls

• visible fails on elements that exist but are hidden. Use attached when you only need the node in the DOM.
• Single-page apps re-render and detach nodes. Re-query the locator after each navigation instead of reusing a stale handle.
• Raising the timeout on a selector that can never match just waits longer. Verify the selector first, raise second.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzFdY7KpKwtcL3U5t_5J0Q