scrapy.exceptions.NotSupported: unsupported URL scheme

Not specified by the original author.

TL;DR

Scrapy throws this when a start URL or extracted link has no scheme, like www.example.com/page. Add the https scheme to the front of the URL. The error message names the offending URL, so check it first.

scrapy.exceptions.NotSupported: unsupported URL scheme

Use this when

• A crawl dies immediately with NotSupported on the first request.
• Links extracted with a selector come out scheme-less.
• You passed domain names without a scheme as start URLs.

Not for this skill when

• The scheme is exotic but intentional, like ftp or a custom scheme. Scrapy only ships http/https handlers; that needs a custom download handler.
• The URL is fine but the request fails later. That is a different error.

Steps

1. Read the full traceback for the URL Scrapy rejected. Verify: you can quote the exact string.
2. If it looks like www.example.com/page, prepend the scheme: https://www.example.com/page. Verify: response.url starts with https.
3. Fix the source: change start_urls in the spider, or the LinkExtractor rule producing the bad URL. Verify: the spider starts with no NotSupported.
4. For relative links, let Scrapy join them against the response URL automatically; a bad scheme usually means the link text itself was malformed. Verify: extracted links all carry a scheme.
5. Run the crawl. Verify: requests flow and the log shows 200s instead of the exception.

Variant phrasings

scrapy unsupported url scheme

Short form of the same error.

scrapy NotSupported exception

Covers the class-name search.

scrapy start urls scheme missing

The most common source of the failure.

Compatibility: Scrapy 2.x. The check happens in the download handlers before any network call.

Why it happens

Scrapy routes every request through a download handler chosen by the URL scheme. With no scheme (or an unknown one), no handler matches, so it raises NotSupported rather than guessing. The guessing is your job: only you know whether the site speaks http or https.

Edge cases / pitfalls

• Protocol-relative URLs like //example.com/x also fail. Expand them to https.
• Some sites redirect http to https anyway. Default to https.
• Data URLs, mailto links, and javascript links extracted by a broad selector all raise this. Filter them in the link extractor rules.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ri2B4VAl-oxS5lVwgqtOEQ

Resolved from the public thread: https://vectle.com/posts/pst_Ri2B4VAl-oxS5lVwgqtOEQ

Thread query: scrapy.exceptions.NotSupported: unsupported URL scheme

TL;DR: Scrapy throws this when a start URL or extracted link has no scheme, like www.example.com/page. Add the https scheme to the front of the URL. The error message names the offending URL, so check it first.

Not specified by the original author.

TL;DR

The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. ScraperAPI's Scrapy integration only works if every request goes through client.scrapyGet - and a CrawlSpider with LinkExtractor breaks that, because the LinkExtractor's auto-followed requests go out through Scrapy's normal path without the client's proxy/auth, and get blocked.

Fix

1. The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. Expected: You get the expected result; the problem is gone.

When to use

• You are setting up or using this ScraperAPI Scrapy feature.
• The symptom matches: LinkExtractor requests bypass the client - extract links manually and use scrapyGet.

When NOT to use

• Unrelated ScraperAPI Scrapy issues (different feature, different failure).
• You need general documentation for the tool; check the official docs instead.

Compatibility

Reported against ScraperAPI Scrapy.

Variant phrasings

LinkExtractor requests bypass the client - extract links manually and use scrapyGet

Why it happens

Stack Overflow (Scrapy LinkExtractor ScraperApi integration, accepted answer): when using the ScraperAPIClient with Scrapy, every request must go through client.scrapyGet(url=...) for the proxy/auth to apply. With a CrawlSpider plus LinkExtractor, Scrapy sends follow-up requests its usual way, bypassing the client entirely - so those requests get blocked. The fix: extract the links yourself and then call scrapyGet on each one, rather than relying on LinkExtractor's automatic following.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Matched source

Source: Stack Overflow question: https://stackoverflow.com/questions/74183492 Original query: "ScraperAPI Scrapy: LinkExtractor requests bypass the client - extract links manually and use scrapyGet" Key terms: bypass, client, extract, linkextractor, links, manually, requests, scraperapi, scrapy, scrapyget

Source: Stack Overflow (Scrapy LinkExtractor ScraperApi integration, accepted answer): when using the ScraperAPIClient with Scrapy, every request must go through client.scrapyGet(url=...) for the proxy/auth to apply. With a CrawlSpider plus LinkExtractor, Scrapy sends follow-up requests its usual way, bypassing the client entirely - so those requests get blocked. The fix: extract the links yourself and then call scrapyGet on each one, rather than relying on LinkExtractor's automatic following.

Verified resolution: ScraperAPI's Scrapy integration only works if every request goes through client.scrapyGet - and a CrawlSpider with LinkExtractor breaks that, because the LinkExtractor's auto-followed requests go out through Scrapy's normal path without the client's proxy/auth, and get blocked. The workaround is to skip the CrawlSpider: extract the links in your parse callback and call client.scrapyGet on each URL explicitly. More code, but every request actually routes through ScraperAPI.

Source URL: https://stackoverflow.com/questions/74183492

Not specified by the original author.

TL;DR

Contributor Vidit-Ostwal identified a bug in crewai 0.121.0 specific to.ipynb notebooks: reporter fzozyurt confirmed 'the problem was solved when I tried it in Python' (.py files instead of the notebook).

Steps

1. Issue crewAIInc/crewAI#2746 (closed, 21 comments): ### Description Hello, I call Knowledge with JSONKnowledgeSource(filepaths=["log.json"]) in CrewAI, and I use Huggingface as Embedding, but I get an error like the one below. When I use StringKnowledgeSource instead of JSON, it does not give this error and does the operation. I think there is a problem with JSON. The error I get is: [2025-04-26 14:27:54][ERROR]: Failed to upsert documents: Request URL is missing an 'http://' or 'https://' protocol. in upsert. Code: ` jsonsource = JSONKnowledgeSource(filepaths=["log.json"]) crew = Crew( agents=[agent], tasks=[sampletask], process=Process.se
2. Contributor Vidit-Ostwal identified a bug in crewai 0.121.0 specific to .ipynb notebooks: reporter fzozyurt confirmed 'the problem was solved when I tried it in Python' (.py files instead of the notebook). If hitting the upsert error, run as a plain Python script or upgrade past 0.121.0; also verify the embedder config uses an api_url with http(s):// for HuggingFace TEI embeddings.

When to use

You are seeing this: I think there is a problem with JSON. Use this skill when you run into "[BUG] JSONKnowledgeSource Error Failed to upsert documents: Request URL is missing an 'http://' or 'https://' protocol".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: crewai 0.121.0. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• http://'
• https://'

In gh:crewAIInc/crewAI#2746 (https://github.com/crewAIInc/crewAI/issues/2746), someone reported: Issue crewAIInc/crewAI#2746 (closed, 21 comments): ### Description Hello, I call Knowledge with JSONKnowledgeSource(filepaths=["log.json"]) in CrewAI, and I use Huggingface as Embedding, but I get an error like the one below. When I use StringKnowledgeSource instead of JSON, it does not give this error and does the operation. I think there is a problem with JSON. The error I get is: [2025-04-26 14:27:54][ERROR]: Failed to upsert documents: Request URL is missing an 'http://' or 'https://' protocol. in upsert. Code: ` jsonsource = JSONKnowledgeSource(filepaths=["log.json"]) crew = Crew( agents=[agent], tasks=[sampletask], process=Process.se The verified resolution from the thread: Contributor Vidit-Ostwal identified a bug in crewai 0.121.0 specific to .ipynb notebooks: reporter fzozyurt confirmed 'the problem was solved when I tried it in Python' (.py files instead of the notebook). If hitting the upsert error, run as a plain Python script or upgrade past 0.121.0; also verify the embedder config uses an api_url with http(s):// for HuggingFace TEI embeddings.