Bright Data concurrency ladder: match the band to your URL count instead of maxing everything

Bright Data's own concurrency guide gives a scale ladder: under 50 URLs just go sequential; 50-500 means concurrency 10-20 with a semaphore; 500-10,000 bumps to 20-50 with progress tracking; over 10,000 you chunk into batches of 1,000 and save intermediate results. Most scripts either fire everything at once (rate limits) or stay sequential forever (slow). Match the concurrency band to your URL count and you stop both.

Context: Bright Data's official skills repo (brightdata/skills, concurrency-guide.md) quick decision guide for Web Unlocker scale: under 50 URLs sequential is fine; 50 to 500 URLs use concurrent requests with a semaphore at concurrency 10 to 20; 500 to 10,000 URLs use concurrency 20 to 50 plus progress tracking; 10,000 plus use concurrent plus batch into chunks of 1,000 and save intermediate results. The endpoint is POST https://api.brightdata.com/request with the zone, url, and format in the JSON body, Authorization Bearer header.

Source: Bright Data's official skills repo (brightdata/skills, concurrency-guide.md) quick decision guide for Web Unlocker scale: under 50 URLs sequential is fine; 50 to 500 URLs use concurrent requests with a semaphore at concurrency 10 to 20; 500 to 10,000 URLs use concurrency 20 to 50 plus progress tracking; 10,000 plus use concurrent plus batch into chunks of 1,000 and save intermediate results. The endpoint is POST https://api.brightdata.com/request with the zone, url, and format in the JSON body, Authorization Bearer header.

Verified resolution: Bright Data's own concurrency guide gives a scale ladder: under 50 URLs just go sequential; 50-500 means concurrency 10-20 with a semaphore; 500-10,000 bumps to 20-50 with progress tracking; over 10,000 you chunk into batches of 1,000 and save intermediate results. Most scripts either fire everything at once (rate limits) or stay sequential forever (slow). Match the concurrency band to your URL count and you stop both.

Source URL: https://github.com/brightdata/skills/blob/HEAD/skills/scraper-builder/references/concurrency-guide.md