Jupyter kernel-environment reproducibility: pin the kernel, lock deps, make notebooks rerun clean

TL;DR

Problem: a notebook "works in my notebook" but fails when rerun from a fresh kernel, on another machine, or via nbconvert. Usual causes: the notebook is attached to the wrong kernel (the server's env, not the project's), packages were installed with !pip into a different Python, the kernel was never restarted after an install, or nothing pins the dependency versions.

When to use

You are seeing this: A Jupyter kernel is a separate process. Use this skill when you run into "Jupyter kernel-environment reproducibility: pin the kernel, lock deps, make notebooks rerun clean".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Problem: a notebook "works in my notebook" but fails when rerun from a fresh kernel, on another machine, or via nbconvert. Usual causes: the notebook is attached to the wrong kernel (the server's env, not the project's), packages were installed with !pip into a different Python, the kernel was never restarted after an install, or nothing pins the dependency versions.

1. Pin the notebook to a real environment

A Jupyter kernel is a separate process. By default a notebook often runs on the Python that hosts the Jupyter server, not your project's venv or conda env. Register your environment as a named kernel:

# Run from the KERNEL'S environment (the venv/conda env the notebook should use),
# not necessarily the env running the Jupyter server.
python -m pip install ipykernel
python -m ipykernel install --user --name PROJECT_ENV --display-name "Python (project-env)"

• --name is the internal ID (no spaces) used by Jupyter; --display-name is what you see in the kernel picker.
• Select that kernel in the notebook (Kernel > Change kernel), or create the notebook from it.
• Inspect and remove stale kernelspecs:

  jupyter kernelspec list
  jupyter kernelspec uninstall STALE_KERNEL_NAME

• Sanity check from inside a cell: import sys; print(sys.executable) should point at your project's Python.

2. Install packages into the kernel's environment

Inside a notebook cell, use the %pip magic, never !pip:

%pip install package-name

• %pip installs into the environment backing the running kernel.
• !pip runs the shell's pip, which may resolve to a different Python (e.g. the notebook server's env). The install "succeeds" but import still fails in the notebook.

3. Restart the kernel after installing anything

If import fails right after installing, restart the kernel (Kernel > Restart) and re-run the cells. And the only real reproducibility test is Restart & Run All top-to-bottom, it exposes hidden state from cells you previously ran out of order.

4. Lock the environment so someone else can recreate it

For pip/venv projects:

pip freeze > requirements.txt   # exact pins of everything installed
pip install -r requirements.txt # recreate elsewhere

For conda:

conda env export > environment.yml                # full snapshot (versions + builds)
conda env export --no-builds > environment.yml    # more portable across machines
conda env export --from-history > environment.yml # only the packages you asked for
conda env create -f environment.yml
conda activate ENV_NAME

• Check requirements.txt / environment.yml in next to the notebook.
• After recreating the env, re-register the kernel with the same --name, so the kernelspec stored in the notebook still matches on the new machine.

5. nbconvert execution gotchas when sharing

Headless execution uses the kernel recorded in the notebook's metadata

jupyter nbconvert --to notebook --execute NOTEBOOK.ipynb

• The kernel name comes from the notebook metadata. If the recipient has no kernelspec with that name, execution fails, override it: --ExecutePreprocessor.kernel_name=PROJECT_ENV.
• Per-cell timeout defaults to 30 seconds; raise it (--ExecutePreprocessor.timeout=600) or disable with -1 for long jobs.
• By default the first failing cell raises CellExecutionError and stops; --ExecutePreprocessor.allow_errors=True records errors in the output and continues.
• Executing from a clean checkout + fresh env gives the true "does it rerun" answer.

Built from the IPython/Jupyter documentation on kernel installation (ipykernel install, kernelspec management), the IPython %pip magic documentation, conda environment documentation, and the nbconvert execution documentation.