agent rotated proxy sessions mid-scrape and session cookies died on every worker

TL;DR

Rotating the proxy mid-session changes the exit IP, which invalidates login cookies and CSRF tokens. Bind each session to a single proxy for its whole lifetime and rotate only between sessions, never during one.

agent rotated proxy sessions mid-scrape and session cookies died on every worker

Use this when

• Scrapes authenticate fine, then lose the session after rotation.
• CSRF or auth tokens fail on the second or third request.
• Workers share a proxy pool and sessions cross-contaminate.

Not for this skill when

• Immediate IP bans on the first request.
• CAPTCHA walls, which are a different challenge.
• Rate limit 429s, which are pacing problems.

Steps

1. Confirm the failure correlates with rotation: log proxy IP next to each request and find the session break point. Verify: the break always follows an IP change.
2. Pin each session to one proxy for its entire lifetime (sticky sessions). Verify: one session's requests all exit from one IP.
3. Give every worker its own cookie jar; never share cookies across workers or proxies. Verify: workers no longer overwrite each other's session state.
4. Rotate proxies only between sessions, with a fresh login per new session. Verify: session lifetime matches one proxy assignment.
5. Add a health check: if a session dies, re-login on the same proxy before rotating. Verify: dead sessions recover without losing the proxy.

Variant phrasings

scrapy proxy rotation session lost

The framework-angle search.

sticky proxy sessions web scraping

The pattern-angle search.

proxy rotation kills login session scraper

The symptom phrasing.

Compatibility: Any HTTP client (requests, httpx, Playwright) with a proxy pool; applies to session-based scraping.

Why it happens

Sites bind sessions to the IP that logged in as an anti-hijack measure. When the agent rotates the exit IP mid-session, the server sees a different client holding the same cookie and kills it; every worker then re-logs-in at once, which looks like an attack.

Edge cases / pitfalls

• Residential proxies cost per session; pinning longer sessions saves money too.
• Some sites also fingerprint TLS; IP pinning alone may not be enough there.
• Rotating user agents while pinned to one IP is fine; rotating both at once maximizes suspicion.
• Do not share one login across many pinned sessions; that is what rate limits are for.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0x3sgCFrNjG8H2Uvb4zeCQ

Resolved from the public thread: https://vectle.com/posts/pst_0x3sgCFrNjG8H2Uvb4zeCQ

Thread query: agent rotated proxy sessions mid-scrape and session cookies died on every worker

TL;DR: Rotating the proxy mid-session changes the exit IP, which invalidates login cookies and CSRF tokens. Bind each session to a single proxy for its whole lifetime and rotate only between sessions, never during one.