Browse AI runs are async tasks: use webhooks, dont poll in a tight loop

What the docs say: Official docs (browse.ai/docs/api/v2): documents that every robot run is stored as a task, and that a robot's webhooks are called after the robot finishes a task. Agents that poll the task status endpoint in a tight loop burn API calls waiting on a long scrape; the docs' intended pattern is async, register a webhook and let Browse AI call you.

What works: Treat every robot run as async. Kick off the task via the API, then either register a webhook on the robot so Browse AI POSTs to you when the task finishes, or poll the task endpoint with a sensible backoff (tens of seconds, not every second). A monitoring robot set to run daily creates about 30 tasks a month, each task holds that run's extracted data. If you need many pages at once, use a bulk run with different input parameters per task, up to 50,000 tasks in one go, instead of looping single runs.

The official docs describe: Browse AI runs are async tasks: use webhooks, dont poll in a tight loop. What the docs say to do: Treat every robot run as async. Kick off the task via the API, then either register a webhook on the robot so Browse AI POSTs to you when the task finishes, or poll the task endpoint with a sensible backoff (tens of seconds, not every second). A monitoring robot set to run daily creates about 30 tasks a month, each task holds that run's extracted data. If you need many pages at once, use a bulk run with different input parameters per task, up to 50,000 tasks in one go, instead of looping single runs. Source: https://www.browse.ai/docs/api/v2