agent could not resume a partial papermill run after a crash, execution failed

TL;DR

Papermill has no built-in resume: a crashed run restarts from zero unless the agent planned for it. Make the agent checkpoint after each expensive cell (write intermediate results to disk), record the last completed cell index durably, and on restart skip to that point or re-run only the tail. The output notebook from the crashed run already shows how far it got; the agent just needs to read it instead of ignoring it.

agent could not resume a partial papermill run after a crash, execution failed

Use this when

• An agent's long papermill run crashes near the end and restarts from zero.
• Re-running the full notebook is expensive (hours).
• The crashed run's output notebook exists with partial results.

Not for this skill when

• The run fails at the same cell every time. Fix the cell, not the resume.
• A human resumes manually. Read the partial output and continue by hand.

Steps

1. Read the crashed output notebook: find the last cell with outputs. Verify: you know the resume index.
2. Add per-cell checkpointing for expensive cells (write results to disk with the cell index in the filename). Verify: checkpoints appear during a run.
3. Record progress durably: the agent writes the last completed index to a state file. Verify: a killed run leaves the state file.
4. Implement resume: on start, load state, skip completed cells (or execute a tail notebook). Verify: a resumed run finishes the tail only.
5. Test with a synthetic crash mid-run. Verify: the resume completes without redoing the head.

Variant phrasings

papermill resume partial run

Short form. Checkpoints plus a state file.

agent resume crashed notebook run

The agent phrasing. Read the partial output.

papermill checkpoint restart

The mechanism phrasing. Expensive cells checkpoint.

Compatibility: papermill 2.x. Resume is agent-side; papermill itself always runs the full notebook.

Why it happens

Papermill's execution model is whole-notebook: it has no resume flag, so agents that treat a crash as 'run it again' pay the full cost every time. The partial output notebook is a free progress record most agents ignore. Checkpointing converts the notebook from an all-or-nothing job into a resumable pipeline, which is what long automated runs need.

Edge cases / pitfalls

• Skipping cells on resume is only valid if they are side-effect free. Audit before skipping.
• Checkpoints must be keyed by inputs, not just cell index. Changed inputs invalidate them.
• A resume that silently uses stale checkpoints produces wrong results confidently. Validate checkpoint freshness.
• Keep the checkpoint logic in the agent, not in the notebook cells. The notebook should stay human-readable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cPGVCtZgft935ZKJIAkwnw