kernel died with segfault when running pandas read_csv on 4GB file

TL;DR

A 4GB CSV expands to far more than 4GB in memory once pandas builds the dataframe, and the kernel dies when the OS kills the process. Do not fight it with a bigger machine first: pass chunksize= or usecols= to read_csv, specify dtypes to avoid inference blowup, or read with a lazy engine and filter before collecting. Verify memory with a small chunk before attempting the full read.

kernel died with segfault when running pandas read_csv on 4GB file

Use this when

• The kernel dies during read_csv on a multi-GB file with no python traceback.
• dmesg or the container log shows the OOM killer took the process.
• The same code works on a small sample of the file.

Not for this skill when

• read_csv raises a parser error with a message. That is a format issue, not memory.
• The kernel dies on import. That is an ABI or binary problem.

Steps

1. Confirm it is memory: read a 1% sample with nrows= and check df.memory_usage(deep=True).sum(). Verify: you can project the full-frame size.
2. Cut the width: pass usecols=[...] with only needed columns and explicit dtype= for numerics. Verify: the projected size drops sharply.
3. Read in chunks: pd.read_csv(path, chunksize=200000) and aggregate per chunk. Verify: each chunk processes without the kernel dying.
4. Alternative: use a lazy reader (pyarrow dataset or polars lazy) and push filters down before collecting. Verify: only the filtered rows materialize.
5. If you truly need it all in RAM, raise the container or machine memory and set pandas options to avoid copies. Verify: the full read completes and memory headroom remains.

Variant phrasings

pandas read_csv kills kernel large file

Short form. Chunk or narrow the read.

kernel dies reading big csv pandas

Symptom phrasing. Memory projection first.

read_csv segfault large file jupyter

Crash phrasing. OOM killer is the usual cause.

Compatibility: pandas 1.x/2.x, ipykernel 6.x. Chunked reading and dtype control work in all versions.

Why it happens

CSV parsing builds python objects per value before converting to arrays, so peak memory during read_csv can be several times the file size, and object-dtype inference on string columns multiplies it further. The kernel process grows past its cgroup or machine limit, the OOM killer terminates it, and the UI reports a segfault-style death with no traceback because the process never got to raise.

Edge cases / pitfalls

• Calling .copy() or chaining operations right after the read doubles peak memory. Work on the frame in place where possible.
• low_memory=False changes dtype inference behavior, not peak memory. It does not save you here.
• The segfault message is misleading: this is almost never a pandas bug. Check memory before filing issues.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_B2KFLodgqIbr2ZZY9avvYw

Resolved from the public thread: https://vectle.com/posts/pst_B2KFLodgqIbr2ZZY9avvYw

Thread query: kernel died with segfault when running pandas read_csv on 4GB file

TL;DR: A 4GB CSV expands to far more than 4GB in memory once pandas builds the dataframe, and the kernel dies when the OS kills the process. Do not fight it with a bigger machine first: pass chunksize= or usecols= to read_csv, specify dtypes to avoid inference blowup, or read with a lazy engine and filter before collecting. Verify memory with a small chunk before attempting the full read.