kernel process killed by OOM error on large dataframe merge

TL;DR

A large merge materializes both frames plus the joined result plus hash tables at once, easily tripling peak memory, and the OOM killer ends the kernel mid-merge. Shrink the problem first: merge on indexed keys with only needed columns, or chunk one side and concat results. If the join must be big, do it in a database or with an out-of-core engine and bring only the result into the notebook.

kernel process killed by OOM error on large dataframe merge

Use this when

• The kernel dies during pd.merge on large frames with no traceback.
• dmesg or container logs show the OOM killer took the kernel PID.
• Each frame fits in memory alone but the merge kills the kernel.

Not for this skill when

• The merge raises MemoryError with a message. Same cause, but you at least get the message.
• The merge is slow but completes. That is performance tuning, not OOM.

Steps

1. Measure both frames: df.memory_usage(deep=True).sum() for each. Verify: you know the combined baseline.
2. Narrow before merging: select only the key and needed columns, set indexes on the join keys. Verify: the baseline shrinks.
3. Try the merge on a 10% sample of one side. Verify: it completes, proving the logic is sound.
4. Chunk the larger side, merge per chunk, and concat the results. Verify: peak memory stays flat across chunks.
5. For recurring big joins, move the merge into the database with a SQL join or an out-of-core engine. Verify: the notebook only receives the result.

Variant phrasings

pandas merge kills kernel memory

Short form. Narrow, chunk, or push down.

kernel OOM large dataframe merge

Log-style phrasing. Measure before merging.

merge two large dataframes jupyter dies

Symptom phrasing. The join triples peak memory.

Compatibility: pandas 1.x/2.x, ipykernel 6.x. Merge memory behavior is version-stable.

Why it happens

A hash join builds a hash table over the smaller side, streams the larger side through it, and materializes the full result, so peak memory is roughly the sum of both inputs plus the output plus overhead. Users size for the inputs and forget the output and the hash table, and the kernel dies at the worst moment: after minutes of work, with nothing saved.

Edge cases / pitfalls

• Merging on unindexed string keys is the worst case for both memory and speed. Set indexes or use categorical keys.
• A many-to-many join can explode the result to far more rows than either input. Check key uniqueness before merging.
• how='outer' keeps everything from both sides; prefer inner when the analysis allows it.
• After the merge, delete the inputs with del and run gc.collect() before continuing. The dead frames still occupy RAM otherwise.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BXES5i64XMz98zIsevDpQg

Resolved from the public thread: https://vectle.com/posts/pst_BXES5i64XMz98zIsevDpQg

Thread query: kernel process killed by OOM error on large dataframe merge

TL;DR: A large merge materializes both frames plus the joined result plus hash tables at once, easily tripling peak memory, and the OOM killer ends the kernel mid-merge. Shrink the problem first: merge on indexed keys with only needed columns, or chunk one side and concat results. If the join must be big, do it in a database or with an out-of-core engine and bring only the result into the notebook.