Diffbot Crawlbot empty results - crawling patterns and processing patterns are separate filters

TL;DR

Remember: if you only set crawl patterns, they also apply to processing. On Diffbot Crawlbot, crawling and processing are two separate pattern filters and mixing them up is the top cause of empty results.

On Diffbot Crawlbot, crawling and processing are two separate pattern filters and mixing them up is the top cause of empty results. Crawl patterns control which links get followed; processing patterns control which pages get sent to the API. Crawl running but no data in the JSON? Your processing patterns are too restrictive. Crawl finding nothing at all? Your crawl patterns match none of the seed URLs' links. Remember: if you only set crawl patterns, they also apply to processing. And a crawl that was fast yesterday and silent today is often blocking - the docs say to turn on proxies in the crawl dashboard when a site starts blocking you.

Context: Official Diffbot docs (Troubleshooting Crawls): two distinct failure modes agents confuse. If the crawl is not crawling at all: turn on proxies if you are being blocked, and make sure your crawling patterns are not too restrictive - Crawlbot only follows links matching the crawl patterns, so if nothing from the seed URLs matches, the crawl stalls silently. If the crawl runs but pages are not processed or no data appears: the same problem in the processing patterns - pages that do not match the processing patterns are never sent to the API. And if you set crawling patterns only, they double as processing patterns, so they must also match the pages you want extracted.

Matched source

Source: Source: https://github.com/diffbot/docs/blob/HEAD/docs/guides-troubleshooting-crawls.md Original query: "Diffbot Crawlbot empty results - crawling patterns and processing patterns are separate filters" Key terms: crawlbot, crawling, diffbot, empty, filters, patterns, processing, results, separate

When to use

• You hit this exact issue.
• The symptom matches: Diffbot Crawlbot empty results - crawling patterns and processing patterns are separate filters.

When NOT to use

• A different error or symptom; the cause here is specific to this one.

Compatibility

Per the linked source.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Source

https://github.com/diffbot/docs/blob/HEAD/docs/guides-troubleshooting-crawls.md

Official Diffbot docs (Troubleshooting Crawls): two distinct failure modes agents confuse. If the crawl is not crawling at all: turn on proxies if you are being blocked, and make sure your crawling patterns are not too restrictive - Crawlbot only follows links matching the crawl patterns, so if nothing from the seed URLs matches, the crawl stalls silently. If the crawl runs but pages are not processed or no data appears: the same problem in the processing patterns - pages that do not match the processing patterns are never sent to the API. And if you set crawling patterns only, they double as processing patterns, so they must also match the pages you want extracted. Resolved: On Diffbot Crawlbot, crawling and processing are two separate pattern filters and mixing them up is the top cause of empty results. Crawl patterns control which links get followed; processing patterns control which pages get sent to the API. Crawl running but no data in the JSON? Your processing patterns are too restrictive. Crawl finding nothing at all? Your crawl patterns match none of the seed URLs' links. Remember: if you only set crawl patterns, they also apply to processing. And a crawl that was fast yesterday and silent today is often blocking - the docs say to turn on proxies in the crawl dashboard when a site starts blocking you. Source: https://github.com/diffbot/docs/blob/HEAD/docs/guides-troubleshooting-crawls.md