agent Playwright session failed mid-authorized-webapp crawl

TL;DR

A resilient crawl checkpoints both the session and the frontier: save the browser storage state after login, persist the URL frontier and visited set to disk as the crawl runs, and on session failure restore the session from storage state and resume from the frontier file. A crawl that cannot resume is a crawl that starts over every time.

agent Playwright session failed mid-authorized-webapp crawl

Use this when

• A Playwright crawl of an authorized app dies partway with session or browser errors.
• The crawl loses its visited set and re-crawls from zero on restart.
• You need crawl state that survives browser crashes.

Not for this skill when

• The crawl never started. That is a launch or login problem, not a resume problem.
• The app blocks automation outright. Then the crawl strategy itself needs rethinking.
• The app is outside your authorized engagement. Do not crawl it.

Steps

1. After a successful login, save the browser storage state to a file. Verify: a fresh browser launched with that file opens the app already logged in.
2. Persist the crawl frontier (queued URLs) and visited set to disk after every page, not in memory. Verify: killing the crawl mid-run leaves files that describe exactly where it stopped.
3. On session failure, do not retry in the dead browser: close it, launch fresh, restore storage state, and verify login with one probe. Verify: the probe shows the authenticated state.
4. Resume from the frontier file, skipping visited URLs. Verify: the resumed crawl visits no duplicates.
5. Cap consecutive session failures: after three, stop and report instead of looping restores. Verify: the run ends with a clear failure report, not a hung loop.

Variant phrasings

playwright crawl session expired resume

Common phrasing.

save and restore playwright browser state

Workflow phrasing.

web app crawl dies halfway no checkpoint

Symptom-level search.

Compatibility: Playwright 1.x storage state. The checkpoint pattern works with any crawl driver.

Why it happens

Crawls are written as one long in-memory loop: the session lives in the browser object and the frontier lives in a Python list. When the browser dies, both vanish, and the agent's only recovery is starting over, which is why long crawls never finish.

Edge cases / pitfalls

• Storage state files contain session cookies: keep them in the engagement workspace with restricted permissions.
• Some apps bind sessions to IP or fingerprint; a restored session may still be rejected. Re-login is the fallback, not a failure.
• Frontier files grow on large apps. Compact them periodically by dropping visited URLs into a separate archive.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_T-EjvKXrV-9QW4ikVKN7cw