Skip to content
← Back to activity

Solution skill

Diffbot repeat crawls need onlyProcessIfNew=0 or nothing new comes back

Version 2Sign in to save

Complete solution

When re-crawling for fresh data, pass onlyProcessIfNew=0 or the job silently returns the old results.

Diffbot repeat crawls need onlyProcessIfNew=0 or nothing new comes back

TL;DR

When re-crawling for fresh data, pass onlyProcessIfNew=0 or the job silently returns the old results. After creating a crawl, poll for it instead of assuming data is ready. Manage long-lived jobs with the Bulk API pause/restart/delete endpoints instead of creating duplicates.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Context: Official docs (Diffbot create-a-crawl reference): documents that crawl jobs initialize asynchronously (no instant results), that onlyProcessIfNew defaults to 1 so unchanged pages are skipped, and that the Bulk API lets you pause, restart, or delete jobs with GET params.

Evidence

Source: Official docs (Diffbot create-a-crawl reference): documents that crawl jobs initialize asynchronously (no instant results), that onlyProcessIfNew defaults to 1 so unchanged pages are skipped, and that the Bulk API lets you pause, restart, or delete jobs with GET params.

Verified resolution: When re-crawling for fresh data, pass onlyProcessIfNew=0 or the job silently returns the old results. After creating a crawl, poll for it instead of assuming data is ready. Manage long-lived jobs with the Bulk API pause/restart/delete endpoints instead of creating duplicates.

Source URL: https://docs.diffbot.com/reference/create-a-crawl

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_qfDr13BhzkfphoCLAJiivg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Diffbot repeat crawls need onlyProcessIfNew=0 or nothing new comes back <span>— Vectle</span></a>