Index has no offsets for WAT/WET records - only WARC supports range fetches

The Common Crawl index only stores offsets and lengths for WARC records, not for WAT or WET files. You cannot range-fetch a single WAT record from the index - you have to download and scan the whole WAT file for the URL you want. The one shortcut: record order is the same in WARC and WAT/WET files, so you can estimate the WAT offset from the WARC offset. If you need precise byte ranges, work with the WARC files instead.

Context: Stack Overflow 45920527 (accepted answer, 5 votes): asked how to get the byte offset and length of a record inside a WAT archive from the Common Crawl index server, to range-download just that record. The accepted answer corrects the premise.

Source: Stack Overflow 45920527 (accepted answer, 5 votes): asked how to get the byte offset and length of a record inside a WAT archive from the Common Crawl index server, to range-download just that record. The accepted answer corrects the premise.

Verified resolution: The Common Crawl index only stores offsets and lengths for WARC records, not for WAT or WET files. You cannot range-fetch a single WAT record from the index - you have to download and scan the whole WAT file for the URL you want. The one shortcut: record order is the same in WARC and WAT/WET files, so you can estimate the WAT offset from the WARC offset. If you need precise byte ranges, work with the WARC files instead.

Source URL: https://stackoverflow.com/questions/45920527