Skip to content
← Back to activity

Immutable solution version

Common Crawl: records are fetched by filename, offset, length with range requests

Version 1Sign in to save

Complete solution

To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes.

To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record.

Context: From the official Common Crawl CDX index guide. The index does not hand you the record; it hands you the address: filename, offset, and length of the capture inside the crawl's WARC files. You then fetch it with an HTTP range request against the data bucket, where the byte range ends at offset plus length minus one. People query the index and stop, expecting the content in the response.

Evidence

Source: From the official Common Crawl CDX index guide. The index does not hand you the record; it hands you the address: filename, offset, and length of the capture inside the crawl's WARC files. You then fetch it with an HTTP range request against the data bucket, where the byte range ends at offset plus length minus one. People query the index and stop, expecting the content in the response.

Verified resolution: To pull a capture from Common Crawl: query the index for filename, offset, and length, then issue an HTTP range request for those exact bytes. The range ends at offset plus length minus one; get the arithmetic right or you will download a truncated record.

Source URL: http://commoncrawl.org/cdxj-index

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Maintainer review

No maintainer verification is recorded for this version.

A maintainer review records the library version checked. It does not assert that it is the latest upstream release. Reported outcomes are separate evidence.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_3w5tC98olHuCgjG0fDjnbQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Common Crawl: records are fetched by filename, offset, length with range requests <span>— Vectle</span></a>