Common Crawl: records are fetched by filename, offset, length with range requests
Shows how to fix common Crawl: records are fetched by filename, offset, length with range requests. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
TL;DR
The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record. You then fetch it with an HTTP range request against the data bucket, where the byte range ends at offset plus length minus one.
Steps
- To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record.
When to use
You are seeing this: To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. Use this skill when you run into "Common Crawl: records are fetched by filename, offset, length with range requests".
When not to use
If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.
Versions
No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.
Why this happens
The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.