papermill output notebook missing executed outputs after crash

TL;DR

Papermill writes the output notebook as it goes, so a crashed run leaves a partial output file with everything up to the failure, unless the crash happened before the first write or corrupted the file. Open the partial output and find the last executed cell; that is your resume point. Fix the failing cell's cause, then re-run the notebook from that cell onward (or re-run fully if the run is cheap) rather than starting blind.

papermill output notebook missing executed outputs after crash

Use this when

• A crashed papermill run left an output notebook with missing or empty outputs.
• You need to know how far the run got before dying.
• Re-running the whole notebook is expensive.

Not for this skill when

• No output file was written at all. The crash happened before execution started.
• Outputs exist but are wrong. That is a logic bug, not a crash artifact.

Steps

1. Open the partial output notebook and find the last cell with outputs. Verify: you know the resume index.
2. Check whether the file is valid: python -c "import nbformat; nbformat.read('out.ipynb',4)". Verify: it parses (a mid-write kill can corrupt it).
3. If corrupt, restore from the input notebook plus your knowledge of progress; do not hand-edit the partial. Verify: you have a clean base.
4. Fix the cause of the crash at the resume point. Verify: the failing cell's issue is addressed.
5. Re-run from the resume point (or fully if cheap). Verify: the new output has complete outputs.

Variant phrasings

papermill crashed output missing

Short form. The partial output is your resume map.

papermill partial output notebook

The artifact phrasing. Find the last executed cell.

papermill output empty after failure

The symptom phrasing. Check how far it got.

Compatibility: papermill 2.x. Progressive output writing is the default behavior.

Why it happens

Papermill streams cell outputs into the output notebook during execution rather than writing once at the end, which is normally a feature: you can watch progress live. On a crash, that same behavior leaves a partial file. Users expecting an all-or-nothing output read the partial file as corruption, but it is actually a precise record of how far the run got.

Edge cases / pitfalls

• A kill -9 mid-write can leave truncated JSON. Validate before trusting the partial file.
• Resuming from the middle only works if earlier cells have no side effects you are skipping. Prefer full re-runs for stateful notebooks.
• The partial output's execution counts reflect the crashed run. Do not treat them as a clean record.
• Automated retries that overwrite the output file destroy the partial evidence. Archive it before retrying.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_15v6RtU65ulOUk1ySS4o3Q