ScraperAPI DataPipeline: recurring schedules need a paid plan, not real time (datapipeline_limits)

ScraperAPI DataPipeline: recurring schedules need a paid plan, not real time (datapipeline_limits)

TL;DR

[ScraperAPI DataPipeline skill reference]: DataPipeline is the scheduled managed option, up to 100,000 items per project with webhook delivery, but it is not real time. If your code needs results now, use the Async API instead. And heads up: recurring schedules need a paid plan, the free tier wont run them.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

If you dont have known URLs yet and need to discover a site, use the Crawler first.

Context: ScraperAPI DataPipeline skill reference: DataPipeline is for scheduled, recurring scraping projects up to 100,000 items with results delivered to a webhook automatically. Do not use it when you need results in real time inside your code; the Async API is programmable while DataPipeline is scheduled. On the free plan, recurring execution is not available: recurring schedules require a paid plan. Use the Crawler instead when exploring a site without known URLs.

Source: ScraperAPI DataPipeline skill reference: DataPipeline is for scheduled, recurring scraping projects up to 100,000 items with results delivered to a webhook automatically. Do not use it when you need results in real time inside your code; the Async API is programmable while DataPipeline is scheduled. On the free plan, recurring execution is not available: recurring schedules require a paid plan. Use the Crawler instead when exploring a site without known URLs.

Verified resolution: [ScraperAPI DataPipeline skill reference]: DataPipeline is the scheduled managed option, up to 100,000 items per project with webhook delivery, but it is not real time. If your code needs results now, use the Async API instead. And heads up: recurring schedules need a paid plan, the free tier wont run them. If you dont have known URLs yet and need to discover a site, use the Crawler first.

Source URL: https://github.com/scraperapi/scraperapi-skills/blob/HEAD/skills/scraperapi-datapipeline/SKILL.md