# Jupyter kernel-environment reproducibility: pin the kernel, lock deps, make notebooks rerun clean

Fix notebooks that work on the author's machine but fail on rerun: register the project's environment as a named kernel, use %pip inside cells instead of !pip, restart the kernel after installs, and export a pinned dependency lockfile so notebooks execute cleanly via nbconvert.

Exact reference: {"kind":"skill_version","skill_id":"skl_jMc91cVvpoVAomkIeBYKCw","version_id":"skv_yBqu0UAfHGxWZ_xX5IwzjQ"}

Applicability: [{"constraint":"ipykernel, nbconvert installed","technology":"Jupyter","version_scheme":"unknown"}]

# Jupyter kernel-environment reproducibility

**Problem:** a notebook "works in my notebook" but fails when rerun from a fresh kernel, on another machine, or via `nbconvert`. Usual causes: the notebook is attached to the wrong kernel (the server's env, not the project's), packages were installed with `!pip` into a different Python, the kernel was never restarted after an install, or nothing pins the dependency versions.

## 1. Pin the notebook to a real environment

A Jupyter *kernel* is a separate process. By default a notebook often runs on the Python that hosts the Jupyter server — not your project's venv or conda env. Register your environment as a named kernel:

```bash
# Run from the KERNEL'S environment (the venv/conda env the notebook should use),
# not necessarily the env running the Jupyter server.
python -m pip install ipykernel
python -m ipykernel install --user --name PROJECT_ENV --display-name "Python (project-env)"
```

- `--name` is the internal ID (no spaces) used by Jupyter; `--display-name` is what you see in the kernel picker.
- Select that kernel in the notebook (Kernel > Change kernel), or create the notebook from it.
- Inspect and remove stale kernelspecs:
  ```bash
  jupyter kernelspec list
  jupyter kernelspec uninstall STALE_KERNEL_NAME
  ```
- Sanity check from inside a cell: `import sys; print(sys.executable)` should point at your project's Python.

## 2. Install packages into the kernel's environment

Inside a notebook cell, use the `%pip` magic — never `!pip`:

```
%pip install package-name
```

- `%pip` installs into the environment backing the *running kernel*.
- `!pip` runs the shell's `pip`, which may resolve to a different Python (e.g. the notebook server's env). The install "succeeds" but `import` still fails in the notebook.

## 3. Restart the kernel after installing anything

If `import` fails right after installing, restart the kernel (Kernel > Restart) and re-run the cells. And the only real reproducibility test is **Restart & Run All** top-to-bottom — it exposes hidden state from cells you previously ran out of order.

## 4. Lock the environment so someone else can recreate it

For pip/venv projects:

```bash
pip freeze > requirements.txt   # exact pins of everything installed
pip install -r requirements.txt # recreate elsewhere
```

For conda:

```bash
conda env export > environment.yml                # full snapshot (versions + builds)
conda env export --no-builds > environment.yml    # more portable across machines
conda env export --from-history > environment.yml # only the packages you asked for
conda env create -f environment.yml
conda activate ENV_NAME
```

- Check `requirements.txt` / `environment.yml` in next to the notebook.
- After recreating the env, re-register the kernel with the *same* `--name`, so the kernelspec stored in the notebook still matches on the new machine.

## 5. nbconvert execution gotchas when sharing

Headless execution uses the kernel recorded in the notebook's metadata:

```bash
jupyter nbconvert --to notebook --execute NOTEBOOK.ipynb
```

- The kernel name comes from the notebook metadata. If the recipient has no kernelspec with that name, execution fails — override it: `--ExecutePreprocessor.kernel_name=PROJECT_ENV`.
- Per-cell timeout defaults to 30 seconds; raise it (`--ExecutePreprocessor.timeout=600`) or disable with `-1` for long jobs.
- By default the first failing cell raises `CellExecutionError` and stops; `--ExecutePreprocessor.allow_errors=True` records errors in the output and continues.
- Executing from a clean checkout + fresh env gives the true "does it rerun" answer.


## Supporting basis and limitations

Built from the IPython/Jupyter documentation on kernel installation (ipykernel install, kernelspec management), the IPython %pip magic documentation, conda environment documentation, and the nbconvert execution documentation.

## Change and rationale

New skill: Jupyter kernel-environment reproducibility (pin kernel, lock deps, clean rerun).

The most common Jupyter support failure is 'works in my notebook but fails when rerun': the notebook is attached to the wrong kernel, packages were installed into a different Python, or nothing pins versions. This skill gives agents a concrete checklist to make notebooks reproducible instead of guessing.