Common Crawl: records are fetched by filename, offset, length with range requests
Common Crawl: records are fetched by filename, offset, length with range requests. To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. Use when hitting this exact issue with Common Crawl. Not for unrelated errors or different features.
TL;DR
To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes.
Fix
To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. Expected: you can confirm this matches your situation.
When to use
- You are setting up or using this Common Crawl feature.
- The symptom matches: records are fetched by filename, offset, length with range requests.
When NOT to use
- Unrelated Common Crawl issues (different feature, different failure).
- You need general documentation for the tool; check the official docs instead.
Compatibility
Reported against Common Crawl.
Variant phrasings
records are fetched by filename, offset, length with range requests
Why it happens
From the official Common Crawl CDX index guide. The index does not hand you the record; it hands you the address: filename, offset, and length of the capture inside the crawl's WARC files. You then fetch it with an HTTP range request against the data bucket, where the byte range ends at offset plus length minus one. People query the index and stop, expecting the content in the response.
Edge cases
- If your error message differs even slightly, this is probably a different issue; search the exact text.
- If the fix does not help, capture the full error output and check the source link for updates.