## TL;DR

The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. ScraperAPI's Scrapy integration only works if every request goes through client.scrapyGet - and a CrawlSpider with LinkExtractor breaks that, because the LinkExtractor's auto-followed requests go out through Scrapy's normal path without the client's proxy/auth, and get blocked.

## Fix

1. The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly.
   Expected: You get the expected result; the problem is gone.

## When to use

- You are setting up or using this ScraperAPI Scrapy feature.
- The symptom matches: LinkExtractor requests bypass the client - extract links manually and use scrapyGet.

## When NOT to use

- Unrelated ScraperAPI Scrapy issues (different feature, different failure).
- You need general documentation for the tool; check the official docs instead.

## Compatibility

Reported against ScraperAPI Scrapy.

## Variant phrasings

### LinkExtractor requests bypass the client - extract links manually and use scrapyGet

## Why it happens

Stack Overflow (Scrapy LinkExtractor ScraperApi integration, accepted answer): when using the ScraperAPIClient with Scrapy, every request must go through client.scrapyGet(url=...) for the proxy/auth to apply. With a CrawlSpider plus LinkExtractor, Scrapy sends follow-up requests its usual way, bypassing the client entirely - so those requests get blocked. The fix: extract the links yourself and then call scrapyGet on each one, rather than relying on LinkExtractor's automatic following.

## Edge cases

- If your error message differs even slightly, this is probably a different issue; search the exact text.
- If the fix does not help, capture the full error output and check the source link for updates.

## Matched source
Source: Stack Overflow question: https://stackoverflow.com/questions/74183492
Original query: "ScraperAPI Scrapy: LinkExtractor requests bypass the client - extract links manually and use scrapyGet"
Key terms: bypass, client, extract, linkextractor, links, manually, requests, scraperapi, scrapy, scrapyget