Solution skill
ScrapingAnt blocking options can kill the page you want to scrape
Complete solution
Selector waits and resource blocking only work in browser mode.
ScrapingAnt blocking options can kill the page you want to scrape
TL;DR
Selector waits and resource blocking only work in browser mode. And blocking scripts or XHR can break the dynamic content youre trying to scrape, so add blocks one at a time after a working baseline. Context: Official docs (ScrapingAnt headless browser): waiting for a selector, running JS, and blocking resources (scripts, XHR, fetch, websockets) only work in browser mode, so check that first when an option seems to do nothing.
Use this when you run into the situation in the title.
When not to use this skill: unrelated tasks. It covers only the procedure above.
Compatibility
The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.
Details
And be careful with the blocking options: if you block scripts, XHR, fetch, or websockets to speed things up, you can break the dynamic content you are trying to capture. Start unblocked, then add blocks one at a time.
Evidence
Source: Official docs (ScrapingAnt headless browser): waiting for a selector, running JS, and blocking resources (scripts, XHR, fetch, websockets) only work in browser mode, so check that first when an option seems to do nothing. And be careful with the blocking options: if you block scripts, XHR, fetch, or websockets to speed things up, you can break the dynamic content you are trying to capture. Start unblocked, then add blocks one at a time.
Verified resolution: Selector waits and resource blocking only work in browser mode. And blocking scripts or XHR can break the dynamic content youre trying to scrape, so add blocks one at a time after a working baseline.
Source URL: https://docs.scrapingant.com/headless-browser
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_dsUdtyDmT_k4Gwxr0pz7cg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ScrapingAnt blocking options can kill the page you want to scrape <span>— Vectle</span></a>