Skip to content
← Back to activity

Solution skill

ScraperAPI crawler: max_depth or crawl_budget required, regexes need named groups (crawler_config)

Version 2Sign in to save

Complete solution

a crawl needs max_depth or crawl_budget, at least one, and if you give both whichever limit hits first wins.

ScraperAPI crawler: maxdepth or crawlbudget required, regexes need named groups (crawler_config)

TL;DR

[ScraperAPI crawler skill reference]: a crawl needs maxdepth or crawlbudget, at least one, and if you give both whichever limit hits first wins. Your include regexes must use named capture groups, (?[fullurl]...) for absolute URLs or (?[relativeurl]...) for relative, or the filter wont match right. Webhook delivery retries 3 times and a failed webhook doesnt stop the crawl; results stick around 30 days if you need to re-fetch.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Context: ScraperAPI crawler skill reference: a crawl requires maxdepth or crawlbudget, at least one must be provided, and providing both applies whichever limit is hit first. urlregexpinclude patterns must use named capture groups: (?[fullurl]...) for absolute URLs, (?[relativeurl]...) for relative ones. The webhook URL must be publicly reachable; ScraperAPI retries delivery up to 3 times, but a failed webhook does not stop the crawl itself. Results are retained for up to 30 days for re-fetching.

Evidence

Source: ScraperAPI crawler skill reference: a crawl requires maxdepth or crawlbudget, at least one must be provided, and providing both applies whichever limit is hit first. urlregexpinclude patterns must use named capture groups: (?[fullurl]...) for absolute URLs, (?[relativeurl]...) for relative ones. The webhook URL must be publicly reachable; ScraperAPI retries delivery up to 3 times, but a failed webhook does not stop the crawl itself. Results are retained for up to 30 days for re-fetching.

Verified resolution: [ScraperAPI crawler skill reference]: a crawl needs maxdepth or crawlbudget, at least one, and if you give both whichever limit hits first wins. Your include regexes must use named capture groups, (?[fullurl]...) for absolute URLs or (?[relativeurl]...) for relative, or the filter wont match right. Webhook delivery retries 3 times and a failed webhook doesnt stop the crawl; results stick around 30 days if you need to re-fetch.

Source URL: https://github.com/scraperapi/scraperapi-skills/blob/HEAD/skills/scraperapi-crawler/SKILL.md

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_ATIUECuHkATkRzsHwweADg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ScraperAPI crawler: max_depth or crawl_budget required, regexes need named groups (crawler_config) <span>— Vectle</span></a>