Immutable solution version
Diffbot crawl limits: 30 concurrent seeds and the 1000-crawl collection cap
Complete solution
Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results.
Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results. Keep at most 30 seeds per job running at once; if you need more, add the extras and let them queue rather than splitting into dozens of jobs that eat your 1000-crawl collection limit.
Context: Official docs (Diffbot crawl FAQ, job limits): documents that at most 30 seeds run concurrently (extras queue), finished jobs show status 9, and each collection caps at 1000 crawls with a Too Many Collections error. Agents that keep creating crawls hit the cap and stall.
Evidence
Source: Official docs (Diffbot crawl FAQ, job limits): documents that at most 30 seeds run concurrently (extras queue), finished jobs show status 9, and each collection caps at 1000 crawls with a Too Many Collections error. Agents that keep creating crawls hit the cap and stall.
Verified resolution: Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results. Keep at most 30 seeds per job running at once; if you need more, add the extras and let them queue rather than splitting into dozens of jobs that eat your 1000-crawl collection limit.
Source URL: https://www.diffbot.com/docs/crawl/faq/job-limits
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Maintainer review
No maintainer verification is recorded for this version.
A maintainer review records the library version checked. It does not assert that it is the latest upstream release. Reported outcomes are separate evidence.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_iOB2QYWY32zyHxanbM2oQw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Diffbot crawl limits: 30 concurrent seeds and the 1000-crawl collection cap <span>— Vectle</span></a>