Skip to content
← Back to activity

Immutable solution version

CheerioCrawler overwrites request cookies when persistCookiesPerSession is false

Version 1Sign in to save

Complete solution

How to fix: CheerioCrawler overwrites request cookies when persistCookiesPerSession is false. The verified fix from the maintainer thread, distilled into reusable steps.

What happened

In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies. In 0.2.x, setting persistCookiesPerSession: false let you manage cookies yourself while still using the session pool. A change in _requestFunction() means only useSessionPool is checked, so the flag is ignored and the crawler's cookies win. The result is that manual cookie headers set on requests get silently replaced.

The fix that worked

This was a regression in the 2.x _requestFunction(): it checked only useSessionPool when deciding whether to apply session cookies, ignoring persistCookiesPerSession: false. The behavior was restored in v2.1, so manual cookie management with a SessionPool works again. Upgrade to v2.1 or later; the maintainer confirmed the fix there.

Evidence

In the thread CheerioCrawler overwrites request cookies when persistCookiesPerSession is false, the question was: In Apify/Crawlee 2.x, a CheerioCrawler configured with a SessionPool (e.g. for proxy rotation) and persistCookiesPerSession: false overwrites manually set request cookies. In 0.2.x, setting persistCookiesPerSession: false let you manage cookies yourself while still using the session pool. A change in _requestFunction() means only useSessionPool is checked, so the flag is ignored and the crawler's cookies win. The result is that manual cookie headers set on requests get silently replaced. The verified resolution: This was a regression in the 2.x _requestFunction(): it checked only useSessionPool when deciding whether to apply session cookies, ignoring persistCookiesPerSession: false. The behavior was restored in v2.1, so manual cookie management with a SessionPool works again. Upgrade to v2.1 or later; the maintainer confirmed the fix there. Original discussion: https://vectle.com/threads/thr_jebh40dqbgncnBYyNM0XWg

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_swKzodCmB1qC3PejK2N6Cg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">CheerioCrawler overwrites request cookies when persistCookiesPerSession is false <span>— Vectle</span></a>