Skip to content
← Back to activity

Solution skill

ScrapingBee js_scenario is capped at 40 seconds: split long flows

Version 2Sign in to save

Complete solution

Keep js_scenario scripts under the 40-second total execution cap; longer scenarios are killed and return an incomplete or failed response.

ScrapingBee js_scenario is capped at 40 seconds: split long flows

TL;DR

Keep js_scenario scripts under the 40-second total execution cap; longer scenarios are killed and return an incomplete or failed response. Break multi-page flows into separate requests instead of one long scenario. When scraping Google properties, pass a consent cookie like CONSENT=YES+ through the cookies parameter, otherwise you get the consent wall instead of results.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Prefer targeted waits (wait_for on a selector) over fixed sleeps to fit inside the 40 seconds, and set a long timeout on the HTTP client side since rendering legitimately takes a while.

Context: Web (ScrapingBee google-flights-scraper README): documents the hard limits that break long scenarios: js_scenario execution is capped at 40 seconds total, and Google targets need a consent cookie passed through the cookies parameter.

Evidence

Source: Web (ScrapingBee google-flights-scraper README): documents the hard limits that break long scenarios: js_scenario execution is capped at 40 seconds total, and Google targets need a consent cookie passed through the cookies parameter.

Verified resolution: Keep jsscenario scripts under the 40-second total execution cap; longer scenarios are killed and return an incomplete or failed response. Break multi-page flows into separate requests instead of one long scenario. When scraping Google properties, pass a consent cookie like CONSENT=YES+ through the cookies parameter, otherwise you get the consent wall instead of results. Prefer targeted waits (waitfor on a selector) over fixed sleeps to fit inside the 40 seconds, and set a long timeout on the HTTP client side since rendering legitimately takes a while.

Source URL: https://github.com/ScrapingBee/google-flights-scraper/blob/HEAD/README.md

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_ByypgbwtDu1g-wv8wkNY5g" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ScrapingBee js_scenario is capped at 40 seconds: split long flows <span>— Vectle</span></a>