CheerioCrawler overwrites request cookies when persistCookiesPerSession is false

TL;DR

Upgrade to v2.1 or later; the maintainer confirmed the fix there. In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies.

Steps

1. In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies. In 0.2.x, setting persistCookiesPerSession: false let you manage cookies yourself while still using the session pool. A change in _requestFunction() means only useSessionPool is checked, so the flag is ignored and the crawler's cookies win. The result is that manual cookie headers set on requests get silently replaced.
2. This was a regression in the 2.x _requestFunction(): it checked only useSessionPool when deciding whether to apply session cookies, ignoring persistCookiesPerSession: false. The behavior was restored in v2.1, so manual cookie management with a SessionPool works again. Upgrade to v2.1 or later; the maintainer confirmed the fix there.

When to use

You are seeing this: In 0.2.x, setting persistCookiesPerSession: false let you manage cookies yourself while still using the session pool. Use this skill when you run into "CheerioCrawler overwrites request cookies when persistCookiesPerSession is false".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v2.1. If you are on something much newer or older, the details may have shifted.

Why this happens

In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies.

In the thread CheerioCrawler overwrites request cookies when persistCookiesPerSession is false, the question was: In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies. In 0.2.x, setting persistCookiesPerSession: false let you manage cookies yourself while still using the session pool. A change in _requestFunction() means only useSessionPool is checked, so the flag is ignored and the crawler's cookies win. The result is that manual cookie headers set on requests get silently replaced. The verified resolution: This was a regression in the 2.x _requestFunction(): it checked only useSessionPool when deciding whether to apply session cookies, ignoring persistCookiesPerSession: false. The behavior was restored in v2.1, so manual cookie management with a SessionPool works again. Upgrade to v2.1 or later; the maintainer confirmed the fix there. Original discussion: https://vectle.com/threads/thr_jebh40dqbgncnBYyNM0XWg