Immutable solution version
ScraperAPI Scrapy: LinkExtractor requests bypass the client - extract links manually and use scrapyGet
Complete solution
ScraperAPI Scrapy: LinkExtractor requests bypass the client - extract links manually and use scrapyGet. The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. Use when hitting this exact issue with ScraperAPI Scrapy. Not for unrelated errors or different features.
TL;DR
The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. ScraperAPI's Scrapy integration only works if every request goes through client.scrapyGet - and a CrawlSpider with LinkExtractor breaks that, because the LinkExtractor's auto-followed requests go out through Scrapy's normal path without the client's proxy/auth, and get blocked.
Fix
The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. Expected: You get the expected result; the problem is gone.
When to use
- You are setting up or using this ScraperAPI Scrapy feature.
- The symptom matches: LinkExtractor requests bypass the client - extract links manually and use scrapyGet.
When NOT to use
- Unrelated ScraperAPI Scrapy issues (different feature, different failure).
- You need general documentation for the tool; check the official docs instead.
Compatibility
Reported against ScraperAPI Scrapy.
Variant phrasings
LinkExtractor requests bypass the client - extract links manually and use scrapyGet
Why it happens
Stack Overflow (Scrapy LinkExtractor ScraperApi integration, accepted answer): when using the ScraperAPIClient with Scrapy, every request must go through client.scrapyGet(url=...) for the proxy/auth to apply. With a CrawlSpider plus LinkExtractor, Scrapy sends follow-up requests its usual way, bypassing the client entirely - so those requests get blocked. The fix: extract the links yourself and then call scrapyGet on each one, rather than relying on LinkExtractor's automatic following.
Edge cases
- If your error message differs even slightly, this is probably a different issue; search the exact text.
- If the fix does not help, capture the full error output and check the source link for updates.
Evidence
Source: Stack Overflow (Scrapy LinkExtractor ScraperApi integration, accepted answer): when using the ScraperAPIClient with Scrapy, every request must go through client.scrapyGet(url=...) for the proxy/auth to apply. With a CrawlSpider plus LinkExtractor, Scrapy sends follow-up requests its usual way, bypassing the client entirely - so those requests get blocked. The fix: extract the links yourself and then call scrapyGet on each one, rather than relying on LinkExtractor's automatic following.
Verified resolution: ScraperAPI's Scrapy integration only works if every request goes through client.scrapyGet - and a CrawlSpider with LinkExtractor breaks that, because the LinkExtractor's auto-followed requests go out through Scrapy's normal path without the client's proxy/auth, and get blocked. The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. More code, but every request actually routes through ScraperAPI.
Source URL: https://stackoverflow.com/questions/74183492
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_tpq2r-zzizJHZQ0jV8_WsA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ScraperAPI Scrapy: LinkExtractor requests bypass the client - extract links manually and use scrapyGet <span>— Vectle</span></a>