Browse AI runs are async tasks: use webhooks, dont poll in a tight loop

TL;DR: What the docs say: Official docs (browse.ai/docs/api/v2): documents that every robot run is stored as a task, and that a robot's webhooks are called after the robot finishes a task. Agents that poll the task status endpoint in a tight loop burn API calls waiting on a long scrape; the docs' intended pattern is async, register a webhook and let Browse AI call you.

Fix

1. What the docs say: Official docs (browse.ai/docs/api/v2): documents that every robot run is stored as a task, and that a robot's webhooks are called after the robot finishes a task. Expected: this specific failure stops.

2. Agents that poll the task status endpoint in a tight loop burn API calls waiting on a long scrape; the docs' intended pattern is async, register a webhook and let Browse AI call you. Expected: this specific failure stops.

3. What works: Treat every robot run as async. Expected: this specific failure stops.

4. A monitoring robot set to run daily creates about 30 tasks a month, each task holds that run's extracted data. Expected: this specific failure stops.

5. If you need many pages at once, use a bulk run with different input parameters per task, up to 50,000 tasks in one go, instead of looping single runs. Expected: this specific failure stops.

Details

Kick off the task via the API, then either register a webhook on the robot so Browse AI POSTs to you when the task finishes, or poll the task endpoint with a sensible backoff (tens of seconds, not every second).

When to use

You hit exactly this: Browse AI runs are async tasks: use webhooks, dont poll in a tight loop in Browse AI.

When not to use

A different error, or the same symptom in a different tool. This page only covers the failure above.

Compatibility

Browse AI (versions mentioned here: v2).

The official docs describe: Browse AI runs are async tasks: use webhooks, dont poll in a tight loop. What the docs say to do: Treat every robot run as async. Kick off the task via the API, then either register a webhook on the robot so Browse AI POSTs to you when the task finishes, or poll the task endpoint with a sensible backoff (tens of seconds, not every second). A monitoring robot set to run daily creates about 30 tasks a month, each task holds that run's extracted data. If you need many pages at once, use a bulk run with different input parameters per task, up to 50,000 tasks in one go, instead of looping single runs. Source: https://www.browse.ai/docs/api/v2