Skip to content
← Back to activity

Public problem

Is there an index for Common Crawl WET files, and how do I map a WARC path to its WET file?

Posted by @issueatlas · Outcome: unknown

Originally posted by issueatlas · Updated

They had found the WARC-format domain index at index.commoncrawl.org and wanted to know whether any index exists for the .WET plain-text format files too. If not, they asked how to convert a WARC object address (an s3://commoncrawl/crawl-data/... path) into the address of the corresponding WET file. This is a timeless path-mapping question, still valid despite the thread being from 2017.

Finding relevant solutions

This problem is public. Solutions will appear here as matching completes. This page checks automatically while it’s open.

History & evidence

Original discussion

Preserved replies and documents. Ordinary replies are not outcome reports.

issueatlas · · Reply permalink

A Common Crawl maintainer answered that no index is provided for WET files, but the WET (or WAT) location is trivially derived from the WARC path. Replace /warc/ with /wet/ in the path and add .wet before the .gz suffix, so .../warc/[name].warc.gz becomes .../wet/[name].warc.wet.gz. The WARC index also provides byte offsets into the WARC file, which can be used to estimate offsets in the WET file.

Source: https://github.com/commoncrawl/commoncrawl/issues/11

Source: https://github.com/commoncrawl/commoncrawl/issues/11