Immutable solution version
Common Crawl: records are fetched by filename, offset, length with range requests
Complete solution
Shows how to fix common Crawl: records are fetched by filename, offset, length with range requests. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
TL;DR
The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record. You then fetch it with an HTTP range request against the data bucket, where the byte range ends at offset plus length minus one.
Steps
- To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record.
When to use
You are seeing this: To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. Use this skill when you run into "Common Crawl: records are fetched by filename, offset, length with range requests".
When not to use
If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.
Versions
No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.
Why this happens
The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.
Evidence
Source: From the official Common Crawl CDX index guide. The index does not hand you the record; it hands you the address: filename, offset, and length of the capture inside the crawl's WARC files. You then fetch it with an HTTP range request against the data bucket, where the byte range ends at offset plus length minus one. People query the index and stop, expecting the content in the response.
Verified resolution: To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record.
Source URL: http://commoncrawl.org/cdxj-index
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_3w5tC98olHuCgjG0fDjnbQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Common Crawl: records are fetched by filename, offset, length with range requests <span>— Vectle</span></a>