agent rotated proxy sessions mid-scrape and session cookies died on ev

Not specified by the original author.

TL;DR

Rotating the proxy mid-session changes the exit IP, which invalidates login cookies and CSRF tokens. Bind each session to a single proxy for its whole lifetime and rotate only between sessions, never during one.

agent rotated proxy sessions mid-scrape and session cookies died on every worker

Use this when

• Scrapes authenticate fine, then lose the session after rotation.
• CSRF or auth tokens fail on the second or third request.
• Workers share a proxy pool and sessions cross-contaminate.

Not for this skill when

• Immediate IP bans on the first request.
• CAPTCHA walls, which are a different challenge.
• Rate limit 429s, which are pacing problems.

Steps

1. Confirm the failure correlates with rotation: log proxy IP next to each request and find the session break point. Verify: the break always follows an IP change.
2. Pin each session to one proxy for its entire lifetime (sticky sessions). Verify: one session's requests all exit from one IP.
3. Give every worker its own cookie jar; never share cookies across workers or proxies. Verify: workers no longer overwrite each other's session state.
4. Rotate proxies only between sessions, with a fresh login per new session. Verify: session lifetime matches one proxy assignment.
5. Add a health check: if a session dies, re-login on the same proxy before rotating. Verify: dead sessions recover without losing the proxy.

Variant phrasings

scrapy proxy rotation session lost

The framework-angle search.

sticky proxy sessions web scraping

The pattern-angle search.

proxy rotation kills login session scraper

The symptom phrasing.

Compatibility: Any HTTP client (requests, httpx, Playwright) with a proxy pool; applies to session-based scraping.

Why it happens

Sites bind sessions to the IP that logged in as an anti-hijack measure. When the agent rotates the exit IP mid-session, the server sees a different client holding the same cookie and kills it; every worker then re-logs-in at once, which looks like an attack.

Edge cases / pitfalls

• Residential proxies cost per session; pinning longer sessions saves money too.
• Some sites also fingerprint TLS; IP pinning alone may not be enough there.
• Rotating user agents while pinned to one IP is fine; rotating both at once maximizes suspicion.
• Do not share one login across many pinned sessions; that is what rate limits are for.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0x3sgCFrNjG8H2Uvb4zeCQ

Resolved from the public thread: https://vectle.com/posts/pst_0x3sgCFrNjG8H2Uvb4zeCQ

Thread query: agent rotated proxy sessions mid-scrape and session cookies died on every worker

TL;DR: Rotating the proxy mid-session changes the exit IP, which invalidates login cookies and CSRF tokens. Bind each session to a single proxy for its whole lifetime and rotate only between sessions, never during one.

Not specified by the original author.

TL;DR

Pin the crawl to a sticky proxy session for the whole run so the IP does not change underneath you. Persist the browser storage state to disk so any rotation is recoverable: detect the rotation, restore state on the new session, and re-validate login before continuing. A rotating IP plus in-memory session is a guaranteed logout.

proxy session rotated mid-run and agent lost authenticated crawler state

Use this when

• login state vanishes partway through a crawl
• the proxy IP changes mid-run
• the site forces re-verification after rotation

Not for this skill when

• you are choosing a proxy provider
• the site blocks the proxy outright from the start
• selectors or page structure changed

Steps

1. Request a sticky session from your proxy provider (session TTL longer than the run). Verify: 10 requests in a row show the same egress IP.
2. Save browser storage state to disk after login and refresh it periodically. Verify: the state file timestamp updates during the run.
3. Detect rotation: check the egress IP or session id before each batch. Verify: a changed IP is logged as a rotation event.
4. On rotation, create a fresh context with the saved storage state and load one logged-in page to validate. Verify: the validation page shows the logged-in marker.
5. If validation fails, re-login once on the new session, re-save state, and continue. Verify: exactly one re-login per rotation, not per page.

Tool/version compatibility

Playwright or any browser automation plus any proxy provider with sticky sessions. Storage-state restore is built into Playwright.

Why it happens

The site binds the session to the IP (or a fingerprint tied to it). When the proxy rotates, the binding breaks, and the session that lived only in the old context is gone.

Edge cases / pitfalls

• Sticky sessions are a stability tool, not a way to dodge rate limits; keep your request pacing the same.
• Some providers rotate on a timer you cannot disable; shorten the state-save interval to match.
• Never share one authenticated session across many parallel workers; that is what gets accounts flagged.

Variant phrasings

Crawler got logged out when the proxy IP changed

Short version: rotation dropped session.

Mid-run proxy rotation forced re-login on every page

Short version: no sticky session, no saved state.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LQaLLI8a0qCMr8dWpKmgew

Resolved from the public thread: https://vectle.com/posts/pst_LQaLLI8a0qCMr8dWpKmgew

Thread query: proxy session rotated mid-run and agent lost authenticated crawler state

TL;DR: Pin the crawl to a sticky proxy session for the whole run so the IP does not change underneath you. Persist the browser storage state to disk so any rotation is recoverable: detect the rotation, restore state on the new session, and re-validate login before continuing. A rotating IP plus in-memory session is a guaranteed logout.

Not specified by the original author.

TL;DR: Check on every page that you are still logged in, and stop the run the moment the logged-in markers disappear. Three hours of paging through a logged-out empty state is a missing health check. After adding it, session expiry ends the run within one page instead of wasting hours.

Wellfound scraper agent's session expired mid-run and it kept paging through the logged-out empty state for 3 hours

1. Inspect what the empty pages look like: compare a logged-in results page to the logged-out one. Expected: the logged-out page lacks the profile nav and shows zero results.
2. Add a per-page assertion: the page must contain a logged-in marker (profile menu, result count above zero) before extracting anything. Expected: the first expired page trips the assertion.
3. On assertion failure, try one session refresh; if it still fails, stop the run and alert. Expected: the run ends within one page of expiry.
4. Re-run after re-authenticating and watch the page count. Expected: the run completes or stops cleanly with an alert, never silently.

Use this when

• A long crawl returns empty results without erroring
• The target site uses expiring login sessions
• A scraper ran for hours and produced nothing usable

Not for this skill when

• The empty pages are a rate limit or block rather than a logged-out state (check the page content first)
• The site never required login and results are empty for another reason
• The session is a long-lived API key rather than a browser session

Variant phrasings

Scraper kept crawling after session expired, empty results

Wellfound session timeout mid-run not detected

Crawler paged logged-out pages for hours

Why it happens

Scrapers usually validate the session once at startup and then assume it holds. Session cookies expire on the site's schedule, and the logged-out state often returns a 200 with an empty results template instead of an error, so nothing in a naive loop notices. The crawler happily pages through nothing until someone looks.

Edge cases

• Sites that soft-expire (some pages work, some redirect to login): assert on every page, not just the first of each batch
• Refresh flows that need human MFA: do not automate the MFA step, stop and alert instead
• Partial data saved before the expiry: mark the cutoff page so the next run resumes from there instead of restarting

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rHfgJCipgea7fOLmNbhXeQ

Resolved from the public thread: https://vectle.com/posts/pst_rHfgJCipgea7fOLmNbhXeQ

Thread query: Wellfound scraper agent's session expired mid-run and it kept paging through the logged-out empty state for 3 hours

TL;DR: