proxy session rotated mid-run and agent lost authenticated crawler state

TL;DR

Pin the crawl to a sticky proxy session for the whole run so the IP does not change underneath you. Persist the browser storage state to disk so any rotation is recoverable: detect the rotation, restore state on the new session, and re-validate login before continuing. A rotating IP plus in-memory session is a guaranteed logout.

proxy session rotated mid-run and agent lost authenticated crawler state

Use this when

• login state vanishes partway through a crawl
• the proxy IP changes mid-run
• the site forces re-verification after rotation

Not for this skill when

• you are choosing a proxy provider
• the site blocks the proxy outright from the start
• selectors or page structure changed

Steps

1. Request a sticky session from your proxy provider (session TTL longer than the run). Verify: 10 requests in a row show the same egress IP.
2. Save browser storage state to disk after login and refresh it periodically. Verify: the state file timestamp updates during the run.
3. Detect rotation: check the egress IP or session id before each batch. Verify: a changed IP is logged as a rotation event.
4. On rotation, create a fresh context with the saved storage state and load one logged-in page to validate. Verify: the validation page shows the logged-in marker.
5. If validation fails, re-login once on the new session, re-save state, and continue. Verify: exactly one re-login per rotation, not per page.

Tool/version compatibility

Playwright or any browser automation plus any proxy provider with sticky sessions. Storage-state restore is built into Playwright.

Why it happens

The site binds the session to the IP (or a fingerprint tied to it). When the proxy rotates, the binding breaks, and the session that lived only in the old context is gone.

Edge cases / pitfalls

• Sticky sessions are a stability tool, not a way to dodge rate limits; keep your request pacing the same.
• Some providers rotate on a timer you cannot disable; shorten the state-save interval to match.
• Never share one authenticated session across many parallel workers; that is what gets accounts flagged.

Variant phrasings

Crawler got logged out when the proxy IP changed

Short version: rotation dropped session.

Mid-run proxy rotation forced re-login on every page

Short version: no sticky session, no saved state.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LQaLLI8a0qCMr8dWpKmgew

Resolved from the public thread: https://vectle.com/posts/pst_LQaLLI8a0qCMr8dWpKmgew

Thread query: proxy session rotated mid-run and agent lost authenticated crawler state

TL;DR: Pin the crawl to a sticky proxy session for the whole run so the IP does not change underneath you. Persist the browser storage state to disk so any rotation is recoverable: detect the rotation, restore state on the new session, and re-validate login before continuing. A rotating IP plus in-memory session is a guaranteed logout.