ScrapingBee render_js defaults on and costs credits: set it false for static pages

ScrapingBee render_js defaults on and costs credits: set it false for static pages

TL;DR

Set render_js=false explicitly whenever you scrape a static page, a PDF, or a file. The parameter defaults to true and each rendered request costs 5 credits, so a bulk job that never needed JavaScript burns budget for nothing. URL-encode the target URL strictly before calling the API: unencoded characters are the most common cause of failed requests and junk responses.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Keep the error map handy: 401 is an invalid API key, 403 is a bad key or blocked account, 429 means you are over your rate limit, 500 is a ScrapingBee-side failure worth retrying. For ordinary targets use premium_proxy=true only when the target blocks datacenter IPs; it is not needed for every request.

Context: Web (proxlane ScrapingBee migration guide): documents two cost gotchas that hit migrators: render_js defaults to true, which bills 5 credits per request, and URL encoding is strict, reportedly the number one support issue for malformed requests.

Source: Web (proxlane ScrapingBee migration guide): documents two cost gotchas that hit migrators: render_js defaults to true, which bills 5 credits per request, and URL encoding is strict, reportedly the number one support issue for malformed requests.

Verified resolution: Set renderjs=false explicitly whenever you scrape a static page, a PDF, or a file. The parameter defaults to true and each rendered request costs 5 credits, so a bulk job that never needed JavaScript burns budget for nothing. URL-encode the target URL strictly before calling the API: unencoded characters are the most common cause of failed requests and junk responses. Keep the error map handy: 401 is an invalid API key, 403 is a bad key or blocked account, 429 means you are over your rate limit, 500 is a ScrapingBee-side failure worth retrying. For ordinary targets use premiumproxy=true only when the target blocks datacenter IPs; it is not needed for every request.

Source URL: https://github.com/proxlane/proxlane/blob/HEAD/apps/web/content/migrate/scrapingbee.md