crawlee session pool: all sessions are blocked or retired

TL;DR

Every session in the pool got blocked or exhausted its retries, so the pool has nothing usable left. Recover by clearing the pool, widening it, and reducing the pressure that burned it: lower concurrency, rotate proxies, and add delays.

crawlee session pool: all sessions are blocked or retired

Use this when

• A Crawlee run aborts with the all-sessions-blocked error.
• A site starts refusing requests or rate-limiting mid-crawl.
• Session counts in the logs climb while usable sessions drop to zero.

Not for this skill when

• Sessions are fine but requests time out. That is a timeout problem.
• You run a single-request script. Session pools only matter for multi-request crawls.

Steps

1. Stop the run and inspect the log: count how many sessions retired versus were blocked. Verify: you know whether it was blocks or natural exhaustion.
2. Clear the persisted pool state: delete the session entries in your configured key-value store (the default storage dir holds them). Verify: the next run starts with fresh sessions.
3. Widen the pool: useSessionPool: true, sessionPoolOptions: { maxPoolSize: 50, sessionOptions: { maxUsageCount: 20 } }. Verify: the pool opens with the new size in the logs.
4. Reduce pressure: set maxConcurrency: 5 and maxRequestRetries: 2, and rotate proxies if you have them configured. Verify: the blocked-session count stays near zero for the first hundred requests.
5. Re-run and watch session health. Verify: the crawl completes instead of aborting.

Variant phrasings

crawlee session pool blocked

Short form of the same error.

all sessions retired crawlee

The exhaustion variant.

crawlee sessions blocked mid crawl

The block-driven variant.

Compatibility: Crawlee 3.x. SessionPool behavior is the same across CheerioCrawler and PlaywrightCrawler.

Why it happens

Crawlee retires a session after too many errors or marks it blocked when the site rejects it. Aggressive concurrency against a protected site burns through the whole pool: every session gets flagged, none are usable, and the crawler gives up instead of hammering a site that has said no.

Edge cases / pitfalls

• Clearing pool state loses logged-in sessions. Re-authenticate after a clear.
• A bigger pool without lower pressure just burns more sessions. Fix the rate first, then widen.
• If the site blocks your IP range outright, no session tuning helps. Use official APIs or licensed data feeds.
• Persisted pool state can resurrect blocked sessions across runs. Clear it when behavior changes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_k9V2xFOqZ93tol3399hcaw

Resolved from the public thread: https://vectle.com/posts/pst_k9V2xFOqZ93tol3399hcaw

Thread query: crawlee session pool: all sessions are blocked or retired

TL;DR: Every session in the pool got blocked or exhausted its retries, so the pool has nothing usable left. Recover by clearing the pool, widening it, and reducing the pressure that burned it: lower concurrency, rotate proxies, and add delays.