Diffbot Crawlbot empty results - crawling patterns and processing patterns are separate filters

TL;DR

Remember: if you only set crawl patterns, they also apply to processing. On Diffbot Crawlbot, crawling and processing are two separate pattern filters and mixing them up is the top cause of empty results.

Fix

1. Remember: if you only set crawl patterns, they also apply to processing. Expected: You get the expected result; the problem is gone.

When to use

• You are dealing with this situation.
• The symptom matches: Diffbot Crawlbot empty results - crawling patterns and processing patterns are separate filters.

When NOT to use

• Unrelated issues with a different cause.
• You need general documentation for the tool; check the official docs instead.

Compatibility

Reported per the linked source.

Variant phrasings

Diffbot Crawlbot empty results - crawling patterns and processing patterns are separate filters

Why it happens

Official Diffbot docs (Troubleshooting Crawls): two distinct failure modes agents confuse. If the crawl is not crawling at all: turn on proxies if you are being blocked, and make sure your crawling patterns are not too restrictive - Crawlbot only follows links matching the crawl patterns, so if nothing from the seed URLs matches, the crawl stalls silently. If the crawl runs but pages are not processed or no data appears: the same problem in the processing patterns - pages that do not match the processing patterns are never sent to the API. And if you set crawling patterns only, they double as processing patterns, so they must also match the pages you want extracted.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Source: Official Diffbot docs (Troubleshooting Crawls): two distinct failure modes agents confuse. If the crawl is not crawling at all: turn on proxies if you are being blocked, and make sure your crawling patterns are not too restrictive - Crawlbot only follows links matching the crawl patterns, so if nothing from the seed URLs matches, the crawl stalls silently. If the crawl runs but pages are not processed or no data appears: the same problem in the processing patterns - pages that do not match the processing patterns are never sent to the API. And if you set crawling patterns only, they double as processing patterns, so they must also match the pages you want extracted.

Verified resolution: On Diffbot Crawlbot, crawling and processing are two separate pattern filters and mixing them up is the top cause of empty results. Crawl patterns control which links get followed; processing patterns control which pages get sent to the API. Crawl running but no data in the JSON? Your processing patterns are too restrictive. Crawl finding nothing at all? Your crawl patterns match none of the seed URLs' links. Remember: if you only set crawl patterns, they also apply to processing. And a crawl that was fast yesterday and silent today is often blocking - the docs say to turn on proxies in the crawl dashboard when a site starts blocking you.

Source URL: https://github.com/diffbot/docs/blob/HEAD/docs/guides-troubleshooting-crawls.md