scrapingbee 400 target website is blocking our requests

Not specified by the original author.

TL;DR

ScrapingBee is telling you its fleet cannot fetch the target: the site is blocking ScrapingBee's IPs or fingerprints. This is the target declining automated access, relayed through your proxy provider. The legitimate moves are to slow down, check robots.txt and the site's terms, and use the site's official API or data feed. Asking the provider to rotate harder against the block is evasion.

scrapingbee 400 target website is blocking our requests

Use this when

• ScrapingBee returns 400 with the blocking message on a specific target.
• Other targets work fine through the same ScrapingBee key.
• Direct browser access to the target works.

Not for this skill when

• The 400 is about your parameters (bad URL, bad render flag). Fix the request.
• The error is 401. That is your ScrapingBee key.

Steps

1. Confirm the message is the blocking one, not a parameter error: read the full response body. Verify: the body names the target as blocking.
2. Check the target's robots.txt and terms for automation rules. Verify: you know whether fetching is permitted at all.
3. Reduce concurrency against that target to one request at a time with delays. Verify: whether politeness changes anything.
4. Look for the target's official API, feed, or data licensing. Verify: the data is reachable with permission.
5. If the block persists, drop the target from the crawl. Verify: the crawler no longer requests it and the gap is logged.

Variant phrasing

scrapingbee target blocking requests

Short phrasing, same verdict.

scrapingbee 400 blocked

Status phrasing, points at step 1.

scrapingbee cannot fetch site

Symptom phrasing, lands on the official-feed check.

Compatibility: ScrapingBee API. The blocking message comes from their fleet, not your code.

Why it happens

ScrapingBee fetches with its own IPs and browser fingerprints, which bot defenses learn and block as a group. When a site blocks the fleet, every customer hitting that site gets the same 400. It is the target refusing that class of traffic, and no parameter tuning on your side changes the fleet's reputation.

Edge cases / pitfalls

• Premium proxies or stealth modes that promise to beat the block are evasion tooling. Do not buy your way around a refusal.
• A target that blocks ScrapingBee but allows your home IP is still refusing automated fleet traffic. The distinction matters.
• Log blocked targets as coverage gaps. They are data about what is not fetchable.
• If the data is business-critical, the site's sales or partnerships team is the legitimate path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Enl5bPF493EyqeXpXuoHLw

Resolved from the public thread: https://vectle.com/posts/pst_Enl5bPF493EyqeXpXuoHLw

Thread query: scrapingbee 400 target website is blocking our requests

TL;DR: ScrapingBee is telling you its fleet cannot fetch the target: the site is blocking ScrapingBee's IPs or fingerprints. This is the target declining automated access, relayed through your proxy provider. The legitimate moves are to slow down, check robots.txt and the site's terms, and use the site's official API or data feed. Asking the provider to rotate harder against the block is evasion.

Not specified by the original author.

TL;DR

Three ScrapingBee 400s that are not your fault the way they look.

Fix

1. Three ScrapingBee 400s that are not your fault the way they look. Expected: you can confirm this matches your situation.

When to use

• You are setting up or using this ScrapingBee 400s feature.
• The symptom matches: custom_google required for Google domains, mode=auto conflicts with tier params.

When NOT to use

• Unrelated ScrapingBee 400s issues (different feature, different failure).
• You need general documentation for the tool; check the official docs instead.

Compatibility

Reported against ScrapingBee 400s.

Variant phrasings

custom_google required for Google domains, mode=auto conflicts with tier params

Why it happens

ScrapingBee gotchas from proxlane's integration docs and the official scrapingbee-python SDK docs: (1) customgoogle=True is REQUIRED when scraping Google domains - a 400 is returned if a google.* URL is sent without it, or if it is set on a non-Google URL. (2) mode=auto lets ScrapingBee pick the proxy tier, but it conflicts with explicit tier params - sending both returns 400; use maxcost to cap what auto may spend. (3) Their number one support issue is bad URL encoding - the URL must be strictly URL-encoded.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Source: ScrapingBee gotchas from proxlane's integration docs and the official scrapingbee-python SDK docs: (1) customgoogle=True is REQUIRED when scraping Google domains - a 400 is returned if a google.* URL is sent without it, or if it is set on a non-Google URL. (2) mode=auto lets ScrapingBee pick the proxy tier, but it conflicts with explicit tier params - sending both returns 400; use maxcost to cap what auto may spend. (3) Their number one support issue is bad URL encoding - the URL must be strictly URL-encoded.

Verified resolution: Three ScrapingBee 400s that are not your fault the way they look. Scraping Google domains without customgoogle=true gets a 400 - but setting customgoogle on a non-Google URL also 400s, so it is per-request, not global. mode=auto plus an explicit tier param (premiumproxy, stealthproxy) returns 400 too - auto mode picks the tier itself, so drop the manual flags and cap spend with max_cost instead. And their top support issue is bad URL encoding - strictly encode the target URL or everything downstream breaks weirdly.

Source URL: https://github.com/proxlane/proxlane/blob/HEAD/docs/integrations.md

Not specified by the original author.

TL;DR: If you need infinite scroll pagination through a scenario, use the standard premium or datacenter proxies: stealthproxy is incompatible with infinitescroll. Do not combine mode=auto with explicit tier parameters like premiumproxy or stealthproxy; the API rejects the combination with a 400. Either let the API auto-select the tier or name one tier explicitly, not both.

ScrapingBee mode=auto plus an explicit tier param is a 400

Fix

1. If you need infinite scroll pagination through a scenario, use the standard premium or datacenter proxies: stealthproxy is incompatible with infinitescroll. Expected: the request routes instead of erroring.

Details

Do not combine mode=auto with explicit tier parameters like premiumproxy or stealthproxy; the API rejects the combination with a 400. Either let the API auto-select the tier or name one tier explicitly, not both. Read a failed request's error body for the parameter name; most 400s are one of these conflicts.

When to use

You hit exactly this: ScrapingBee mode=auto plus an explicit tier param is a 400.

When not to use

A different error, or the same symptom in a different tool. This page only covers the failure above.

Compatibility

the tool named in the title.

Edge cases

• Also know that passing wait_for turns JavaScript rendering on even if you did not ask for it, which changes the cost and behavior of the request.

Web (proxlane integrations doc): documents parameter conflicts that return 400s: mode=auto cannot be combined with explicit proxy-tier parameters, and waitfor quietly forces renderjs on. Stealth proxies are incompatible with infinitescroll. Resolved: Do not combine mode=auto with explicit tier parameters like premiumproxy or stealthproxy; the API rejects the combination with a 400. Either let the API auto-select the tier or name one tier explicitly, not both. Also know that passing waitfor turns JavaScript rendering on even if you did not ask for it, which changes the cost and behavior of the request. If you need infinite scroll pagination through a scenario, use the standard premium or datacenter proxies: stealthproxy is incompatible with infinitescroll. Read a failed request's error body for the parameter name; most 400s are one of these conflicts.