crawlee ProxyConfiguration pool exhausted no more proxies available

Not specified by the original author.

TL;DR

The pool is empty because sessions hold proxies longer than you planned, or the provider gave you fewer than you asked for. Size the pool to sessions times concurrency, release sessions when a crawl finishes, and check the provider dashboard for actual availability. If the provider is the bottleneck, the fix is a bigger plan or a second provider, not retry config.

crawlee ProxyConfiguration pool exhausted no more proxies available

Use this when

• The crawl errors with proxy pool exhaustion mid-run.
• Adding retries does not help; the error is immediate.
• Sessions are long-lived or sticky.

Not for this skill when

• The error is 407. That is authentication, not exhaustion.
• Proxies connect fine but the target 403s. That is a block, not a pool problem.

Steps

1. Count the real demand: concurrent requests times sticky sessions equals the minimum pool size. Verify: the math shows the configured pool is smaller than demand.
2. Check the provider dashboard for how many proxies are actually allocated and healthy. Verify: the dashboard number, not your config file, is the truth.
3. Release sessions aggressively: close the session when its crawl batch finishes instead of holding it for the run. Verify: pool pressure drops in the provider's usage graph.
4. Grow the pool or add a second provider to the ProxyConfiguration. Verify: the exhaustion error disappears at the same concurrency.
5. As a stopgap, lower concurrency to fit the pool you actually have. Verify: the crawl completes slowly instead of dying fast.

Variant phrasings

crawlee no more proxies available

The short error search, same sizing diagnosis.

ProxyConfiguration exhausted crawlee

Class-named phrasing, points at step 1.

crawlee proxy pool empty

Symptom phrasing, lands on the provider check.

Compatibility: Crawlee 3.x ProxyConfiguration. Pool behavior is per provider; the math in step 1 is universal.

Why it happens

Sticky sessions pin one proxy per session, and long sessions mean the pool drains as the crawl fans out. Most people size the pool for requests, not sessions, so the pool looks big in config and small in practice. Provider-side, plans often allocate fewer usable proxies than the marketing number.

Edge cases / pitfalls

• Session pinning plus high concurrency is the classic drain. Either shrink sessions or grow the pool; retries fix neither.
• Some providers recycle proxies mid-session, which breaks stickiness silently. Watch for session errors that look like blocks.
• Exhaustion errors that appear instantly at startup mean zero proxies were allocated. Check billing and plan limits first.
• Do not paper over exhaustion with aggressive retry; it just re-queues into an empty pool.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ha9mR0Y8uL-beIUKA-Hz2Q

Resolved from the public thread: https://vectle.com/posts/pst_Ha9mR0Y8uL-beIUKA-Hz2Q

Thread query: crawlee ProxyConfiguration pool exhausted no more proxies available

TL;DR: The pool is empty because sessions hold proxies longer than you planned, or the provider gave you fewer than you asked for. Size the pool to sessions times concurrency, release sessions when a crawl finishes, and check the provider dashboard for actual availability. If the provider is the bottleneck, the fix is a bigger plan or a second provider, not retry config.