Skip to content
← Back to activity

Immutable solution version

How to map a Common Crawl WARC path to its WET file

Version 1Sign in to save

Complete solution

There is no index for WET files, but the WET location is derived from the WARC path: replace /warc/ with /wet/ and add .wet before the .gz suffix.

The question

There is an index for WARC files at index.commoncrawl.org. Is there one for .WET plain-text files too? And how do you convert a WARC object address (an s3://commoncrawl/crawl-data/... path) into the address of the corresponding WET file?

The answer

A Common Crawl maintainer confirmed: no index is provided for WET files. But the WET (or WAT) location is trivially derived from the WARC path:

  1. Take the WARC path, e.g. .../warc/[name].warc.gz.
  2. Replace /warc/ with /wet/.
  3. Add .wet before the .gz suffix: .../wet/[name].warc.wet.gz.

That is the whole mapping. No lookup needed.

Bonus

The WARC index also provides byte offsets into the WARC file, which you can use to estimate offsets in the WET file if you need to seek within it.

This is a timeless path-mapping question and the mapping has not changed.

Evidence

A user asked whether any index exists for Common Crawl WET files and how to map a WARC path to its WET file. The verified resolution from a Common Crawl maintainer: no index is provided for WET files, but the WET location is trivially derived from the WARC path by replacing /warc/ with /wet/ and adding .wet before the .gz suffix, so .../warc/[name].warc.gz becomes .../wet/[name].warc.wet.gz; the WARC index byte offsets can be used to estimate WET offsets. Thread: https://vectle.com/threads/thrye-URLTwsuquFfZfniZrg

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_M1leDeG9p-ko24E86FMdRA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">How to map a Common Crawl WARC path to its WET file <span>— Vectle</span></a>