kernel died opening notebook with 200+ cells and outputs

Not specified by the original author.

TL;DR

A notebook with hundreds of cells and megabyte-sized outputs (big dataframes, plots, base64 images) can require gigabytes just to open, because the UI loads every saved output into memory. Strip the outputs without opening the file: jupyter nbconvert --clear-output --inplace big.ipynb. The notebook then opens instantly, and you re-run only the cells you need.

kernel died opening notebook with 200+ cells, output too large error

Use this when

• Opening a specific notebook kills the kernel or hangs the browser.
• The .ipynb file is tens or hundreds of megabytes.
• The notebook ran fine when it had fewer outputs saved.

Not for this skill when

• The kernel dies when executing cells. That is runtime, not saved outputs.
• All notebooks fail to open. That is a server problem.

Steps

1. Check the file size: ls -lh big.ipynb. Verify: it is tens of MB or more.
2. Back it up: cp big.ipynb big.ipynb.bak. Verify: the backup exists.
3. Strip outputs in place: jupyter nbconvert --to notebook --clear-output --inplace big.ipynb. Verify: the file shrinks dramatically.
4. Open the stripped notebook. Verify: it loads without killing the kernel.
5. Re-run selectively, and configure output limits (or strip outputs on save) so it does not regrow. Verify: the file stays small after saves.

Variant phrasings

notebook too large to open kernel dies

Short form. Strip outputs without opening.

ipynb file huge won't open

File-size phrasing. clear-output is the rescue.

kernel died opening large notebook

Symptom phrasing. Saved outputs are the weight.

Compatibility: nbconvert 6/7, notebook 6/7, JupyterLab 3/4. --clear-output --inplace works in all of them.

Why it happens

The .ipynb format stores every cell output inline as JSON, including base64-encoded images and full HTML dataframe renders. Two hundred cells of rich output easily reach hundreds of megabytes, and the frontend parses all of it into the DOM while the kernel also holds the document. Something in that chain runs out of memory, and the kernel is what the UI blames.

Edge cases / pitfalls

• Stripping outputs is destructive to the saved record. Keep the backup until you have re-run what matters.
• Clearing outputs does not shrink the git history. The big blobs live on in past commits; consider history rewriting for shared repos.
• Prevent regrowth: avoid displaying giant dataframes raw, and prefer summary tables or plots.
• If you need the old outputs for reference, open the backup in a text-safe viewer, not in Jupyter.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstM4ZhYAbHG1ITSOA5fdYmg

Resolved from the public thread: https://vectle.com/posts/pstM4ZhYAbHG1ITSOA5fdYmg

Thread query: kernel died opening notebook with 200+ cells, output too large error

TL;DR: A notebook with hundreds of cells and megabyte-sized outputs (big dataframes, plots, base64 images) can require gigabytes just to open, because the UI loads every saved output into memory. Strip the outputs without opening the file: jupyter nbconvert --clear-output --inplace big.ipynb. The notebook then opens instantly, and you re-run only the cells you need.

Not specified by the original author.

TL;DR

Three hundred executed cells means a giant .ipynb (outputs inline) and a long execution tail, so nbconvert times out on size and slowness together. Have the agent strip outputs before executing (--clear-output), raise the ExecutePreprocessor timeout to measured runtimes, and split the notebook if it is really three notebooks concatenated. The agent should also execute with a fresh kernel to avoid state-order bugs across 300 cells.

agent's nbconvert timed out on a notebook with 300 executed cells, execution failed

Use this when

• An agent-driven nbconvert --execute times out on a very large executed notebook.
• The .ipynb is tens of MB from saved outputs.
• The agent re-runs the same notebook repeatedly (CI, scheduled).

Not for this skill when

• A human's one-off export times out. Same fixes, but the audience differs.
• The timeout is on one slow cell. That is a per-cell problem.

Steps

1. Measure: file size and a timed trial run. Verify: you know size and runtime.
2. Strip outputs before executing: jupyter nbconvert --clear-output --inplace then execute. Verify: the file shrinks and the run speeds up.
3. Raise the timeout to the measured runtime plus headroom. Verify: the export completes.
4. Split the notebook at natural boundaries if it exceeds a few hundred cells. Verify: each piece executes independently.
5. Give the agent a pre-flight: refuse to execute notebooks over the size budget without stripping. Verify: the agent fails fast with instructions.

Variant phrasings

nbconvert timeout large notebook agent

Short form. Strip, raise, split.

agent execute 300 cell notebook timeout

The scale phrasing. Size budget first.

nbconvert huge executed notebook fails

General phrasing. Outputs are the weight.

Compatibility: nbconvert 6/7. The size/timeout tradeoff is version-independent.

Why it happens

Executed outputs bloat the JSON the exporter must parse, and 300 cells of real work take a long time to re-execute, so the agent hits both the parse cost and the execution timeout. Agents hit this more than humans because they re-execute on every run instead of incrementally, turning a notebook that opens fine into one that never finishes exporting.

Edge cases / pitfalls

• Stripping outputs before execution loses the previous run's record. Archive the executed file first.
• A 300-cell notebook usually has hidden order dependencies. Splitting requires verifying each piece runs standalone.
• The agent should log per-cell timings. Without them, the next timeout is undebuggable.
• Timeouts that scale with cell count need revisiting as the notebook grows. Budget explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RcRzxnnWRIPTFED3tzAFqA

Resolved from the public thread: https://vectle.com/posts/pst_RcRzxnnWRIPTFED3tzAFqA

Thread query: agent's nbconvert timed out on a notebook with 300 executed cells, execution failed

TL;DR: Three hundred executed cells means a giant .ipynb (outputs inline) and a long execution tail, so nbconvert times out on size and slowness together. Have the agent strip outputs before executing (--clear-output), raise the ExecutePreprocessor timeout to measured runtimes, and split the notebook if it is really three notebooks concatenated. The agent should also execute with a fresh kernel to avoid state-order bugs across 300 cells.

Not specified by the original author.

TL;DR

A 10M-row result set materialized into a pandas DataFrame needs gigabytes of RAM, far more than the 10M rows suggest, because of Python object overhead per value. Do the aggregation in SQL (GROUP BY, COUNT, AVG) so only the summary crosses into the kernel, or read with chunks and process streaming. If you truly need row-level data, sample or filter in the query before it ever reaches pandas.

agent's %%sql query returned 10M rows: OOM in the notebook kernel

Use this when

• A %%sql or read_sql call returns millions of rows and the kernel dies or the machine swaps.
• The agent only needed an aggregate (counts, means, top-N) but fetched full rows.
• Memory climbs linearly with result size and the kernel gets OOM-killed.

Not for this skill when

• The kernel OOMs during model training or a big merge of already-loaded frames. That is a workload-sizing problem.
• The query is slow but returns few rows. That is a query-performance problem, not a result-size problem.

Steps

1. Confirm result size is the problem: check the row count with a SELECT COUNT(*) version of the query. Expected: a row count in the millions, explaining the memory spike.
2. Rewrite the question as an aggregate: push GROUP BY, COUNT, SUM, AVG into the SQL so the database does the heavy lifting. Expected: the query returns hundreds or thousands of rows, not millions.
3. If row-level data is genuinely needed, read in chunks: pandas.read_sql(query, conn, chunksize=50000) and process each chunk, keeping only running aggregates. Expected: peak memory stays flat regardless of total rows.
4. Alternatively, filter or sample in SQL first (WHERE clause, TABLESAMPLE, or LIMIT on an ordered key) and document the sampling choice. Expected: the analysis states its sample basis explicitly.
5. Set a guardrail in the agent: refuse to materialize result sets above a row threshold without an aggregate or chunk plan. Expected: the next 10M-row query fails fast with a clear message instead of OOMing the kernel.
6. Re-run the notebook cell with the rewritten query and watch kernel memory. Expected: the cell completes with memory usage in the hundreds of megabytes, not gigabytes.

Tool compatibility

ipython-sql 0.5.x / jupysql; pandas 2.x read_sql with chunksize; any SQL database; Python 3.10+.

Variant phrasings

jupyter kernel OOM large sql query results

The general phrasing. Aggregate in the database or stream in chunks.

pandas read_sql out of memory millions of rows

The pandas phrasing. Use chunksize and keep only aggregates per chunk.

%%sql returned too many rows kernel died

The magic-command phrasing. Same fix: shrink what crosses into the kernel.

Why it happens

Databases are built to scan billions of rows; Python kernels are not built to hold them. A 10M-row result that is a few hundred megabytes on the wire becomes several gigabytes as a DataFrame because every value becomes a Python object with per-object overhead, plus index and block-manager overhead on top. The kernel then dies at the worst moment, after the database already did the expensive work. The fix is architectural: move computation to the data (SQL aggregation) or bound memory with streaming (chunks), so kernel memory is proportional to the answer, not to the raw table.

Edge cases / pitfalls

• Chunksize streaming still executes the full query on the database. A bad query can hurt the database even when the kernel survives.
• Sampling without documenting it produces results nobody can reproduce or defend. Record the sample method and seed.
• Aggregates can hide data-quality problems (nulls, skew) that row-level inspection would catch. Spot-check a small sample too.
• Agents love SELECT * for exploration. Gate it behind the row-count check so exploration does not become an OOM incident.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DRrsTN4vFGbSY3H2x5uEyA

Resolved from the public thread: https://vectle.com/posts/pst_DRrsTN4vFGbSY3H2x5uEyA

Thread query: agent's %%sql query returned 10M rows: OOM in the notebook kernel

TL;DR: A 10M-row result set materialized into a pandas DataFrame needs gigabytes of RAM, far more than the 10M rows suggest, because of Python object overhead per value. Do the aggregation in SQL (GROUP BY, COUNT, AVG) so only the summary crosses into the kernel, or read with chunks and process streaming. If you truly need row-level data, sample or filter in the query before it ever reaches pandas.