Immutable solution version
Small test corpus - one random WARC file is already a random sample
Complete solution
Small test corpus - one random WARC file is already a random sample: For a small test corpus, download just one random WARC, WAT, or WET file.
For a small test corpus, download just one random WARC, WAT, or WET file. Grab the file list for any monthly crawl, unzip it, pick one line at random, prefix with https://data.commoncrawl.org/, and download - one file is tens to hundreds of MB and is a random sample by itself. Need more data? Pick more files at random. There is no need to process a whole crawl for experiments.
Context: Stack Overflow 55762306 (accepted answer, 12 votes): asked how to grab a tiny sample (tens of MB) of Common Crawl for IR tests without sifting hundreds of GB. The accepted answer: every single WARC file is already a random sample.
Matched source
Source: Published skill Original query: "Small test corpus - one random WARC file is already a random sample" Key terms: already, corpus, file, random, sample, small, test, warc
Evidence
Source: Stack Overflow 55762306 (accepted answer, 12 votes): asked how to grab a tiny sample (tens of MB) of Common Crawl for IR tests without sifting hundreds of GB. The accepted answer: every single WARC file is already a random sample.
Verified resolution: For a small test corpus, download just one random WARC, WAT, or WET file. Grab the file list for any monthly crawl, unzip it, pick one line at random, prefix with https://data.commoncrawl.org/, and download - one file is tens to hundreds of MB and is a random sample by itself. Need more data? Pick more files at random. There is no need to process a whole crawl for experiments.
Source URL: https://stackoverflow.com/questions/55762306
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_tIcM1mC1XQLzsCbqRZI1YQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Small test corpus - one random WARC file is already a random sample <span>— Vectle</span></a>