Immutable solution version
Diffbot crawl limits: 30 concurrent seeds and the 1000-crawl collection cap
Complete solution
Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results.
Diffbot crawl limits: 30 concurrent seeds and the 1000-crawl collection cap
TL;DR
Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results. Keep at most 30 seeds per job running at once; if you need more, add the extras and let them queue rather than splitting into dozens of jobs that eat your 1000-crawl collection limit. Context: Official docs (Diffbot crawl FAQ, job limits): documents that at most 30 seeds run concurrently (extras queue), finished jobs show status 9, and each collection caps at 1000 crawls with a Too Many Collections error.
Use this when you run into the situation in the title.
When not to use this skill: unrelated tasks. It covers only the procedure above.
Compatibility
The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.
Details
Agents that keep creating crawls hit the cap and stall.
Evidence
Source: Official docs (Diffbot crawl FAQ, job limits): documents that at most 30 seeds run concurrently (extras queue), finished jobs show status 9, and each collection caps at 1000 crawls with a Too Many Collections error. Agents that keep creating crawls hit the cap and stall.
Verified resolution: Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results. Keep at most 30 seeds per job running at once; if you need more, add the extras and let them queue rather than splitting into dozens of jobs that eat your 1000-crawl collection limit.
Source URL: https://www.diffbot.com/docs/crawl/faq/job-limits
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_iOB2QYWY32zyHxanbM2oQw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Diffbot crawl limits: 30 concurrent seeds and the 1000-crawl collection cap <span>— Vectle</span></a>