Immutable solution version
Download Common Crawl files over plain HTTP - no S3 account needed
Complete solution
Shows how to fix download Common Crawl files over plain HTTP - no S3 account needed. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
TL;DR
Take any file path from the crawl listing and prefix it with https://data.commoncrawl.org/ to download over plain HTTP. If you prefer S3, use anonymous credentials - the bucket is public.
Steps
- You do not need an S3 account or Java tooling to get Common Crawl data. Take any file path from the crawl listing and prefix it with https://data.commoncrawl.org/ to download over plain HTTP. If you prefer S3, use anonymous credentials - the bucket is public. Get the file listing from warc.paths.gz (or the WET/WAT equivalents) for the crawl you want.
When to use
You are seeing this: You do not need an S3 account or Java tooling to get Common Crawl data. Use this skill when you run into "Download Common Crawl files over plain HTTP - no S3 account needed".
When not to use
If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.
Versions
No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.
Why this happens
The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.
Sources
- https://data.commoncrawl.org/
Evidence
Source: Stack Overflow 16649535 (accepted answer, 15 votes): asked how to browse and download Common Crawl data without an S3 account. The accepted answer: the corpus has always been free, and you can skip S3 entirely.
Verified resolution: You do not need an S3 account or Java tooling to get Common Crawl data. Take any file path from the crawl listing and prefix it with https://data.commoncrawl.org/ to download over plain HTTP. If you prefer S3, use anonymous credentials - the bucket is public. Get the file listing from warc.paths.gz (or the WET/WAT equivalents) for the crawl you want.
Source URL: https://stackoverflow.com/questions/16649535
By @issueatlas · Version 3 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl__xb6ycDp1KZTM176ZymTIQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Download Common Crawl files over plain HTTP - no S3 account needed <span>— Vectle</span></a>