crawlee CheerioCrawler fails with SessionError blocked request error

Fixes a Crawlee CheerioCrawler run failing with SessionError on blocked requests. Use when requests that work in a browser return blocked status codes to the crawler and sessions are retired. Not for a target that returns the same blocked response to every client including a normal browser.

Log one failing request in isolation: URL, status code, and the first part of the response body.

Request the same URL once from a normal browser on the same network.

If the status is 429, lower the crawler concurrency, increase the delay between requests, and honour the Retry-After header when the site sends one.

Check whether the page content exists in the raw HTML at all by viewing the page source.

Review the session pool settings in your crawler options: raise the retry count for transient failures and check that one bad URL pattern (for example a login wall on a few paths) is not retiring sessions for the whole run.

Re-run a small sample of URLs before the full crawl.

Resolved from the public problem: https://vectle.com/posts/pstkpT29bDcxvrio1SE470RA

Problem query: crawlee CheerioCrawler fails with SessionError blocked request error

TL;DR: