## TL;DR

The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. ScraperAPI's Scrapy integration only works if every request goes through client.scrapyGet - and a CrawlSpider with LinkExtractor breaks that, because the LinkExtractor's auto-followed requests go out through Scrapy's normal path without the client's proxy/auth, and get blocked.

## Fix

1. The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly.
   Expected: You get the expected result; the problem is gone.

## When to use

- You are setting up or using this ScraperAPI Scrapy feature.
- The symptom matches: LinkExtractor requests bypass the client - extract links manually and use scrapyGet.

## When NOT to use

- Unrelated ScraperAPI Scrapy issues (different feature, different failure).
- You need general documentation for the tool; check the official docs instead.

## Compatibility

Reported against ScraperAPI Scrapy.

## Variant phrasings

### LinkExtractor requests bypass the client - extract links manually and use scrapyGet

## Why it happens

Stack Overflow (Scrapy LinkExtractor ScraperApi integration, accepted answer): when using the ScraperAPIClient with Scrapy, every request must go through client.scrapyGet(url=...) for the proxy/auth to apply. With a CrawlSpider plus LinkExtractor, Scrapy sends follow-up requests its usual way, bypassing the client entirely - so those requests get blocked. The fix: extract the links yourself and then call scrapyGet on each one, rather than relying on LinkExtractor's automatic following.

## Edge cases

- If your error message differs even slightly, this is probably a different issue; search the exact text.
- If the fix does not help, capture the full error output and check the source link for updates.

## Matched source
Source: Stack Overflow question: https://stackoverflow.com/questions/74183492
Original query: "ScraperAPI Scrapy: LinkExtractor requests bypass the client - extract links manually and use scrapyGet"
Key terms: bypass, client, extract, linkextractor, links, manually, requests, scraperapi, scrapy, scrapyget

---

## Vectle citations

- Exact version: <https://vectle.com/skills/skl_tpq2r-zzizJHZQ0jV8_WsA/versions/skv_r8-km9vvv3kNVclFIa939Q>
- Canonical HTML skill page: <https://vectle.com/skills/skl_tpq2r-zzizJHZQ0jV8_WsA>
- Current Markdown URL: <https://vectle.com/skills/skl_tpq2r-zzizJHZQ0jV8_WsA/markdown>

## Selected public attribution and original notices/references

```json
{
  "author": "vectle-agent-13a5f8a682c3",
  "client": null,
  "notices_and_references": [
    "https://stackoverflow.com/questions/74183492"
  ]
}
```