Index has no offsets for WAT/WET records - only WARC supports range fetches

TL;DR

The one shortcut: record order is the same in WARC and WAT/WET files, so you can estimate the WAT offset from the WARC offset. If you need precise byte ranges, work with the WARC files instead.

Steps

1. The Common Crawl index only stores offsets and lengths for WARC records, not for WAT or WET files. You cannot range-fetch a single WAT record from the index - you have to download and scan the whole WAT file for the URL you want. The one shortcut: record order is the same in WARC and WAT/WET files, so you can estimate the WAT offset from the WARC offset. If you need precise byte ranges, work with the WARC files instead.

When to use

You are seeing this: The Common Crawl index only stores offsets and lengths for WARC records, not for WAT or WET files. Use this skill when you run into "Index has no offsets for WAT/WET records - only WARC supports range fetches".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Stack Overflow 45920527 (accepted answer, 5 votes): asked how to get the byte offset and length of a record inside a WAT archive from the Common Crawl index server, to range-download just that record. The accepted answer corrects the premise.

Verified resolution: The Common Crawl index only stores offsets and lengths for WARC records, not for WAT or WET files. You cannot range-fetch a single WAT record from the index - you have to download and scan the whole WAT file for the URL you want. The one shortcut: record order is the same in WARC and WAT/WET files, so you can estimate the WAT offset from the WARC offset. If you need precise byte ranges, work with the WARC files instead.

Source URL: https://stackoverflow.com/questions/45920527