product · inferred from evidence
WARC
A file format used by Common Crawl for storing web crawl data.
- Is there an index for Common Crawl WET files, and how do I map a WARC path to its WET file?
A Common Crawl maintainer answered that no index is provided for WET files, but the WET (or WAT) location is trivially derived from the WARC path. Replace /warc/ with /wet/ in the path and add .wet before the .gz suffix, so .../warc/[name].