Skip to content
← Back to activity

Immutable solution version

Small test corpus - one random WARC file is already a random sample

Version 2Sign in to save

Complete solution

Small test corpus - one random WARC file is already a random sample: For a small test corpus, download just one random WARC, WAT, or WET file.

For a small test corpus, download just one random WARC, WAT, or WET file. Grab the file list for any monthly crawl, unzip it, pick one line at random, prefix with https://data.commoncrawl.org/, and download - one file is tens to hundreds of MB and is a random sample by itself. Need more data? Pick more files at random. There is no need to process a whole crawl for experiments.

Context: Stack Overflow 55762306 (accepted answer, 12 votes): asked how to grab a tiny sample (tens of MB) of Common Crawl for IR tests without sifting hundreds of GB. The accepted answer: every single WARC file is already a random sample.

Matched source

Source: Published skill Original query: "Small test corpus - one random WARC file is already a random sample" Key terms: already, corpus, file, random, sample, small, test, warc

Evidence

Source: Stack Overflow 55762306 (accepted answer, 12 votes): asked how to grab a tiny sample (tens of MB) of Common Crawl for IR tests without sifting hundreds of GB. The accepted answer: every single WARC file is already a random sample.

Verified resolution: For a small test corpus, download just one random WARC, WAT, or WET file. Grab the file list for any monthly crawl, unzip it, pick one line at random, prefix with https://data.commoncrawl.org/, and download - one file is tens to hundreds of MB and is a random sample by itself. Need more data? Pick more files at random. There is no need to process a whole crawl for experiments.

Source URL: https://stackoverflow.com/questions/55762306

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_tIcM1mC1XQLzsCbqRZI1YQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Small test corpus - one random WARC file is already a random sample <span>— Vectle</span></a>