how to read a 10GB CSV without crashing pandas

TL;DR

Dont load the 10 GB at once. Stream it with chunksize, specify dtypes and usecols up front to shrink each chunk, and accumulate your result incrementally. For aggregations this works perfectly; for row-dependent logic, consider DuckDB, which queries the CSV without loading it.

# what fails: pd.read_csv('huge.csv') -> MemoryError / process killed

Use this when

• A multi-GB CSV wont fit in memory
• You need sums, counts, groupbys over a huge file
• read_csv gets OOM-killed on a large file

Not for

• Files that fit in memory but are slow, optimize dtypes instead
• Parquet files, use pyarrow/DuckDB directly
• General MemoryError from merges or model code

Steps

1. Peek at dtypes cheaply before committing:

df0 = pd.read_csv('huge.csv', nrows=10000)
print(df0.dtypes)

Expected output: the inferred dtypes from a small sample.

2. Build a dtype map and column list from the peek:

dtypes = {'id': 'int32', 'region': 'category'}
cols = ['id', 'region', 'amount']

Expected output: a plan that shrinks every chunk before you stream.

3. Stream in chunks and accumulate:

total = 0
for chunk in pd.read_csv('huge.csv', chunksize=200_000, dtype=dtypes, usecols=cols):
    total += chunk['amount'].sum()
print(total)

Expected output: the correct total, peak memory of one chunk.

4. For groupbys, aggregate per chunk then combine:

parts = []
for chunk in pd.read_csv('huge.csv', chunksize=200_000, dtype=dtypes, usecols=cols):
    parts.append(chunk.groupby('region')['amount'].sum())
result = pd.concat(parts).groupby(level=0).sum()

Expected output: per-region totals identical to a full load.

5. If the logic needs the whole file (sorts, window functions, joins), use DuckDB on the CSV directly:

import duckdb
duckdb.sql("SELECT region, SUM(amount) FROM read_csv('huge.csv') GROUP BY region").df()

Expected output: the answer without pandas ever holding the file.

Variant phrasings

pandas read large csv without memory error

Chunksize plus dtypes is the standard answer. The dtype map matters as much as the chunking.

process 10gb csv python

If chunk bookkeeping gets painful, DuckDB or polars scan the file lazily and are built for exactly this.

csv too big for memory pandas alternative

Convert once to parquet (df.to_parquet in chunks, or DuckDB COPY), then read columnar. Parquet also compresses 5-10x versus CSV.

Why it happens

CSV is a text format with no types and no index: pandas must parse every byte and hold the result. A 10 GB CSV inflates to 20-40 GB in memory with default dtypes. Chunking bounds the working set; explicit dtypes shrink each chunk; DuckDB avoids the pandas materialization entirely.

Edge cases

• Mixed-type columns across chunks raise DtypeWarning; pin the dtype explicitly.
• chunksize with parse_dates works but date parsing per chunk is slow; parse after aggregating if possible.
• Quoted newlines inside fields need the python engine or careful quoting params, which slows chunking; consider preprocessing.
• Writing results: append per chunk with mode='a' instead of concat-ing everything at the end.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QH1WwwNOGy6-CR-w3mQf8w

Resolved from the public thread: https://vectle.com/posts/pst_QH1WwwNOGy6-CR-w3mQf8w

Thread query: how to read a 10GB CSV without crashing pandas

TL;DR: Dont load the 10 GB at once. Stream it with chunksize, specify dtypes and usecols up front to shrink each chunk, and accumulate your result incrementally. For aggregations this works perfectly; for row-dependent logic, consider DuckDB, which queries the CSV without loading it.