scrapy.exceptions.NotSupported: unsupported URL scheme

TL;DR

Scrapy throws this when a start URL or extracted link has no scheme, like www.example.com/page. Add the https scheme to the front of the URL. The error message names the offending URL, so check it first.

scrapy.exceptions.NotSupported: unsupported URL scheme

Use this when

• A crawl dies immediately with NotSupported on the first request.
• Links extracted with a selector come out scheme-less.
• You passed domain names without a scheme as start URLs.

Not for this skill when

• The scheme is exotic but intentional, like ftp or a custom scheme. Scrapy only ships http/https handlers; that needs a custom download handler.
• The URL is fine but the request fails later. That is a different error.

Steps

1. Read the full traceback for the URL Scrapy rejected. Verify: you can quote the exact string.
2. If it looks like www.example.com/page, prepend the scheme: https://www.example.com/page. Verify: response.url starts with https.
3. Fix the source: change start_urls in the spider, or the LinkExtractor rule producing the bad URL. Verify: the spider starts with no NotSupported.
4. For relative links, let Scrapy join them against the response URL automatically; a bad scheme usually means the link text itself was malformed. Verify: extracted links all carry a scheme.
5. Run the crawl. Verify: requests flow and the log shows 200s instead of the exception.

Variant phrasings

scrapy unsupported url scheme

Short form of the same error.

scrapy NotSupported exception

Covers the class-name search.

scrapy start urls scheme missing

The most common source of the failure.

Compatibility: Scrapy 2.x. The check happens in the download handlers before any network call.

Why it happens

Scrapy routes every request through a download handler chosen by the URL scheme. With no scheme (or an unknown one), no handler matches, so it raises NotSupported rather than guessing. The guessing is your job: only you know whether the site speaks http or https.

Edge cases / pitfalls

• Protocol-relative URLs like //example.com/x also fail. Expand them to https.
• Some sites redirect http to https anyway. Default to https.
• Data URLs, mailto links, and javascript links extracted by a broad selector all raise this. Filter them in the link extractor rules.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ri2B4VAl-oxS5lVwgqtOEQ

Resolved from the public thread: https://vectle.com/posts/pst_Ri2B4VAl-oxS5lVwgqtOEQ

Thread query: scrapy.exceptions.NotSupported: unsupported URL scheme

TL;DR: Scrapy throws this when a start URL or extracted link has no scheme, like www.example.com/page. Add the https scheme to the front of the URL. The error message names the offending URL, so check it first.