crawlee PlaywrightCrawler navigation timeout exceeded
Fixes PlaywrightCrawler navigation timeouts in Crawlee via navigationTimeoutSecs, resource blocking, and concurrency tuning. Use when crawls fail on goto with navigation timeout exceeded on slow or heavy pages. Not for block pages, login walls, or post-navigation selector timeouts.
TL;DR
Raise navigationTimeoutSecs on the PlaywrightCrawler (default 30 seconds) and cut page weight by blocking resources you do not need. Slow pages, heavy third-party scripts, and pages that never settle all trip the default.
crawlee PlaywrightCrawler navigation timeout exceededUse this when
- Crawls fail with "navigation timeout exceeded" on goto.
- Pages load slowly or pull in heavy third-party scripts.
- The page works in a manual browser but times out in the crawler.
Not for this skill when
- The page never loads because of a block or login wall. A longer timeout will not fix a 403 or a challenge page.
- The timeout happens after navigation, on a selector wait. That is a selector problem.
Steps
- Reproduce on one URL with crawler logging on. Verify: the error names navigation, not a selector.
- Set
navigationTimeoutSecs: 60in the crawler options. Verify: the same URL now completes. - Block heavy resources you do not need: in
preNavigationHooks, usepage.routeto abort image, font, and analytics requests. Verify: page weight drops and timeouts get rarer. - For chronically slow sites, combine a 120s navigation timeout with lower
maxConcurrency. Verify: a full crawl run finishes with no navigation timeouts.
Variant phrasings
playwrightcrawler timeout
Short form of the same failure.
crawlee navigation timeout
General search for the same error.
goto timeout crawlee
The Playwright-level symptom.
Compatibility: Crawlee 3.x with the Playwright plugin. navigationTimeoutSecs is in seconds; Playwright's own timeouts are in milliseconds, so do not mix the units.
Why it happens
PlaywrightCrawler waits for the page's navigation to settle before handing it to your handler. The default 30-second cap assumes a normal page; pages with slow ad networks, streaming scripts, or giant payloads blow past it.
Edge cases / pitfalls
- Some pages never settle because of long-polling scripts. Blocking those requests fixes it; a bigger number does not.
- A bigger timeout on a blocked page just waits 60 seconds to show you a block page. Check the status code first.
- If a challenge or block page appears, treat it as access not granted. Do not try to evade it; look for an official API or data feed instead.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_0P4ZPRBRNPF-nR4jSsZXZw