Immutable solution version
Find a specific archived URL - query the columnar index with Athena, then range-fetch
Complete solution
Find a specific archived URL - query the columnar index with Athena, then range-fetch: Do not download the index files to find one URL.
Do not download the index files to find one URL. Query the Common Crawl columnar index with AWS Athena like a SQL table to find your URL, which gives you the filename, byte offset, and length. Then fetch just that byte range from the WARC file over HTTP with a Range request. That turns a multi-gigabyte scan into two cheap requests.
Context: Stack Overflow 38869741 (accepted answer, 4 votes): asked how to find whether a specific URL is archived in Common Crawl without downloading whole index files. The accepted answer points at the columnar index queryable with SQL.
Matched source
Source: Published skill Original query: "Find a specific archived URL - query the columnar index with Athena, then range-fetch" Key terms: archived, athena, columnar, fetch, find, index, query, range, specific, then
Evidence
Source: Stack Overflow 38869741 (accepted answer, 4 votes): asked how to find whether a specific URL is archived in Common Crawl without downloading whole index files. The accepted answer points at the columnar index queryable with SQL.
Verified resolution: Do not download the index files to find one URL. Query the Common Crawl columnar index with AWS Athena like a SQL table to find your URL, which gives you the filename, byte offset, and length. Then fetch just that byte range from the WARC file over HTTP with a Range request. That turns a multi-gigabyte scan into two cheap requests.
Source URL: https://stackoverflow.com/questions/38869741
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_Vx9X6ACJIyOpskk1-7_1cA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Find a specific archived URL - query the columnar index with Athena, then range-fetch <span>— Vectle</span></a>