Immutable solution version
CheerioCrawler overwrites request cookies when persistCookiesPerSession is false
Complete solution
How to fix: CheerioCrawler overwrites request cookies when persistCookiesPerSession is false. The verified fix from the maintainer thread, distilled into reusable steps.
What happened
In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies. In 0.2.x, setting persistCookiesPerSession: false let you manage cookies yourself while still using the session pool. A change in _requestFunction() means only useSessionPool is checked, so the flag is ignored and the crawler's cookies win. The result is that manual cookie headers set on requests get silently replaced.
The fix that worked
This was a regression in the 2.x _requestFunction(): it checked only useSessionPool when deciding whether to apply session cookies, ignoring persistCookiesPerSession: false. The behavior was restored in v2.1, so manual cookie management with a SessionPool works again. Upgrade to v2.1 or later; the maintainer confirmed the fix there.
Evidence
In the thread CheerioCrawler overwrites request cookies when persistCookiesPerSession is false, the question was: In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies. In 0.2.x, setting persistCookiesPerSession: false let you manage cookies yourself while still using the session pool. A change in _requestFunction() means only useSessionPool is checked, so the flag is ignored and the crawler's cookies win. The result is that manual cookie headers set on requests get silently replaced. The verified resolution: This was a regression in the 2.x _requestFunction(): it checked only useSessionPool when deciding whether to apply session cookies, ignoring persistCookiesPerSession: false. The behavior was restored in v2.1, so manual cookie management with a SessionPool works again. Upgrade to v2.1 or later; the maintainer confirmed the fix there. Original discussion: https://vectle.com/threads/thr_jebh40dqbgncnBYyNM0XWg
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_swKzodCmB1qC3PejK2N6Cg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">CheerioCrawler overwrites request cookies when persistCookiesPerSession is false <span>— Vectle</span></a>