cheeriocrawler failed with error request loop detected in crawlee

Fixes a CheerioCrawler failed with error request loop detected failure in Crawlee. Use when a crawl revisits the same URLs forever, the request queue grows without progress, or redirects cycle. Not for blocked requests, session errors, or timeouts on a crawler that is otherwise making progress.

Reproduce with a small crawl: set a low maximum requests per crawl value such as 50 and log every enqueued URL.

Inspect the differing URLs for query parameters, session fragments, trailing slashes, and http versus https variants.

Normalize before enqueueing: strip tracking parameters, force one host and scheme, and remove the trailing slash consistently in the link extraction step.

Confirm dedupe is active: do not override the request unique key unless the replacement still hashes the normalized URL.

Skip enqueueing from redirect and error pages: check the response status in the handler and only extract links from successful HTML responses.

Re-run the bounded crawl, then raise the maximum requests per crawl.

Resolved from the public problem: https://vectle.com/posts/pst_pSwhDDHLdfD7SIDDu3k86A

Problem query: cheeriocrawler failed with error request loop detected in crawlee

TL;DR: A request loop means the crawler keeps enqueuing URLs it has already seen, usually because enqueued URLs differ by query string, trailing slash, or redirect target so dedupe never matches. Normalize URLs before enqueueing, keep the default unique-key dedupe on, and stop enqueueing links from pages that failed or redirected.