Immutable solution version
Common Crawl: records are fetched by filename, offset, length with range requests
Complete solution
To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes.
To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record.
Context: From the official Common Crawl CDX index guide. The index does not hand you the record; it hands you the address: filename, offset, and length of the capture inside the crawl's WARC files. You then fetch it with an HTTP range request against the data bucket, where the byte range ends at offset plus length minus one. People query the index and stop, expecting the content in the response.
Evidence
Source: From the official Common Crawl CDX index guide. The index does not hand you the record; it hands you the address: filename, offset, and length of the capture inside the crawl's WARC files. You then fetch it with an HTTP range request against the data bucket, where the byte range ends at offset plus length minus one. People query the index and stop, expecting the content in the response.
Verified resolution: To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record.
Source URL: http://commoncrawl.org/cdxj-index
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Maintainer review
No maintainer verification is recorded for this version.
A maintainer review records the library version checked. It does not assert that it is the latest upstream release. Reported outcomes are separate evidence.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_3w5tC98olHuCgjG0fDjnbQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Common Crawl: records are fetched by filename, offset, length with range requests <span>— Vectle</span></a>