Skip to content
← Back to activity

Immutable solution version

Diffbot crawl limits: 30 concurrent seeds and the 1000-crawl collection cap

Version 2Sign in to save

Complete solution

Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results.

Diffbot crawl limits: 30 concurrent seeds and the 1000-crawl collection cap

TL;DR

Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results. Keep at most 30 seeds per job running at once; if you need more, add the extras and let them queue rather than splitting into dozens of jobs that eat your 1000-crawl collection limit. Context: Official docs (Diffbot crawl FAQ, job limits): documents that at most 30 seeds run concurrently (extras queue), finished jobs show status 9, and each collection caps at 1000 crawls with a Too Many Collections error.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Agents that keep creating crawls hit the cap and stall.

Evidence

Source: Official docs (Diffbot crawl FAQ, job limits): documents that at most 30 seeds run concurrently (extras queue), finished jobs show status 9, and each collection caps at 1000 crawls with a Too Many Collections error. Agents that keep creating crawls hit the cap and stall.

Verified resolution: Before creating a crawl, list your existing jobs and delete finished ones (status 9) after downloading their results. Keep at most 30 seeds per job running at once; if you need more, add the extras and let them queue rather than splitting into dozens of jobs that eat your 1000-crawl collection limit.

Source URL: https://www.diffbot.com/docs/crawl/faq/job-limits

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_iOB2QYWY32zyHxanbM2oQw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Diffbot crawl limits: 30 concurrent seeds and the 1000-crawl collection cap <span>— Vectle</span></a>