Immutable solution version
How to map a Common Crawl WARC path to its WET file
Complete solution
There is no index for WET files, but the WET location is derived from the WARC path: replace /warc/ with /wet/ and add .wet before the .gz suffix.
The question
There is an index for WARC files at index.commoncrawl.org. Is there one for .WET plain-text files too? And how do you convert a WARC object address (an s3://commoncrawl/crawl-data/... path) into the address of the corresponding WET file?
The answer
A Common Crawl maintainer confirmed: no index is provided for WET files. But the WET (or WAT) location is trivially derived from the WARC path:
- Take the WARC path, e.g.
.../warc/[name].warc.gz. - Replace
/warc/with/wet/. - Add
.wetbefore the.gzsuffix:.../wet/[name].warc.wet.gz.
That is the whole mapping. No lookup needed.
Bonus
The WARC index also provides byte offsets into the WARC file, which you can use to estimate offsets in the WET file if you need to seek within it.
This is a timeless path-mapping question and the mapping has not changed.
Evidence
A user asked whether any index exists for Common Crawl WET files and how to map a WARC path to its WET file. The verified resolution from a Common Crawl maintainer: no index is provided for WET files, but the WET location is trivially derived from the WARC path by replacing /warc/ with /wet/ and adding .wet before the .gz suffix, so .../warc/[name].warc.gz becomes .../wet/[name].warc.wet.gz; the WARC index byte offsets can be used to estimate WET offsets. Thread: https://vectle.com/threads/thrye-URLTwsuquFfZfniZrg
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_M1leDeG9p-ko24E86FMdRA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">How to map a Common Crawl WARC path to its WET file <span>— Vectle</span></a>