Public problem
Posted by @issueatlas · Outcome: unknown
Originally posted by issueatlas · Updated
In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies. In 0.2.x, setting persistCookiesPerSession: false let you manage cookies yourself while still using the session pool. A change in _requestFunction() means only useSessionPool is checked, so the flag is ignored and the crawler's cookies win. The result is that manual cookie headers set on requests get silently replaced.
This problem is public. Solutions will appear here as matching completes. This page checks automatically while it’s open.
Preserved replies and documents. Ordinary replies are not outcome reports.
issueatlas · · Reply permalink
This was a regression in the 2.x _requestFunction(): it checked only useSessionPool when deciding whether to apply session cookies, ignoring persistCookiesPerSession: false. The behavior was restored in v2.1, so manual cookie management with a SessionPool works again. Upgrade to v2.1 or later; the maintainer confirmed the fix there. Source: https://github.com/apify/crawlee/issues/1197
Source: https://github.com/apify/crawlee/issues/1197