ScraperAPI Scrapy: LinkExtractor requests bypass the client - extract links manually and use scrapyGet

TL;DR

The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. ScraperAPI's Scrapy integration only works if every request goes through client.scrapyGet - and a CrawlSpider with LinkExtractor breaks that, because the LinkExtractor's auto-followed requests go out through Scrapy's normal path without the client's proxy/auth, and get blocked.

Fix

1. The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. Expected: You get the expected result; the problem is gone.

When to use

• You are setting up or using this ScraperAPI Scrapy feature.
• The symptom matches: LinkExtractor requests bypass the client - extract links manually and use scrapyGet.

When NOT to use

• Unrelated ScraperAPI Scrapy issues (different feature, different failure).
• You need general documentation for the tool; check the official docs instead.

Compatibility

Reported against ScraperAPI Scrapy.

Variant phrasings

LinkExtractor requests bypass the client - extract links manually and use scrapyGet

Why it happens

Stack Overflow (Scrapy LinkExtractor ScraperApi integration, accepted answer): when using the ScraperAPIClient with Scrapy, every request must go through client.scrapyGet(url=...) for the proxy/auth to apply. With a CrawlSpider plus LinkExtractor, Scrapy sends follow-up requests its usual way, bypassing the client entirely - so those requests get blocked. The fix: extract the links yourself and then call scrapyGet on each one, rather than relying on LinkExtractor's automatic following.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Matched source

Source: Stack Overflow question: https://stackoverflow.com/questions/74183492 Original query: "ScraperAPI Scrapy: LinkExtractor requests bypass the client - extract links manually and use scrapyGet" Key terms: bypass, client, extract, linkextractor, links, manually, requests, scraperapi, scrapy, scrapyget

Source: Stack Overflow (Scrapy LinkExtractor ScraperApi integration, accepted answer): when using the ScraperAPIClient with Scrapy, every request must go through client.scrapyGet(url=...) for the proxy/auth to apply. With a CrawlSpider plus LinkExtractor, Scrapy sends follow-up requests its usual way, bypassing the client entirely - so those requests get blocked. The fix: extract the links yourself and then call scrapyGet on each one, rather than relying on LinkExtractor's automatic following.

Verified resolution: ScraperAPI's Scrapy integration only works if every request goes through client.scrapyGet - and a CrawlSpider with LinkExtractor breaks that, because the LinkExtractor's auto-followed requests go out through Scrapy's normal path without the client's proxy/auth, and get blocked. The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. More code, but every request actually routes through ScraperAPI.

Source URL: https://stackoverflow.com/questions/74183492