agent ran the notebook in CI without pinning ipykernel: version drift crash

TL;DR

An unpinned pip install ipykernel in CI floats to the newest release, and a new ipykernel (or its new traitlets, jupyterclient) can break notebook execution that worked yesterday. Pin ipykernel, nbclient, nbformat, and `jupyterclient` in the CI requirements and rebuild from a lockfile. Then re-run the notebook in CI to confirm the pinned stack executes it cleanly.

agent ran the notebook in CI without pinning ipykernel: version drift crash

Use this when

• A notebook that passed CI last week fails now with no code changes, after an ipykernel or dependency upgrade.
• The CI log shows a fresh pip install ipykernel pulling a new major or minor version.
• An agent generates the CI environment without a lockfile.

Not for this skill when

• The failure is a labextension vs JupyterLab version conflict. That is the Lab 4 extension problem, a different fix.
• The kernel starts but user packages are missing. That is an incomplete requirements file, not version drift.

Steps

1. Find the drift: compare the CI log's installed versions (pip freeze output in the log) against the last green run. Expected: you can name the package and version that changed.
2. Pin the execution stack in the CI requirements: ipykernel, nbclient, nbformat, jupyter_client, traitlets, each with == pins. Expected: the requirements file names exact versions.
3. Better, generate a lockfile from a known-good environment and have CI install from it. Expected: CI installs byte-identical dependencies on every run.
4. Clear the CI cache for the Python environment so the next run builds from the pins, not from a stale cached layer. Expected: the build log shows the pinned versions installing.
5. Re-run the notebook execution step in CI. Expected: the notebook executes cleanly with the pinned stack.
6. Add a CI step that fails when the lockfile is out of sync with the requirements, so the next drift is caught at PR time. Expected: version changes become deliberate PRs, not silent breakage.

Tool compatibility

ipykernel 6.x; nbclient 0.10.x; nbformat 5.x; jupyter_client 8.x; pip or conda CI environments.

Variant phrasings

ipykernel version drift broke CI notebook

The drift phrasing. Pin the execution stack and install from a lockfile.

notebook CI failed after ipykernel upgrade

The upgrade phrasing. The fix is pins, not chasing the new version's changes.

agent CI notebook unpinned dependencies crash

The agent phrasing. Agents that write CI configs must emit pins, not bare package names.

Why it happens

The notebook execution stack is several tightly-coupled packages, and ipykernel's major releases regularly change message-protocol details, asyncio handling, and dependency floors. CI that installs unpinned packages re-resolves the world on every run, so a passing pipeline can break overnight with zero code changes, and the failure looks like a notebook bug. Pinning converts dependency changes from accidents into reviewed decisions. A lockfile goes further by freezing the transitive closure, which is where the real surprises (a new traitlets or jupyter_client) usually come from.

Edge cases / pitfalls

• Pinning only ipykernel still leaves jupyter_client and traitlets floating. Pin the whole execution stack or use a lockfile.
• Stale Docker layer caches can keep installing the old unpinned set even after you add pins. Bust the cache on requirements changes.
• A lockfile generated on one platform may not install on another (macOS vs Linux wheels). Generate the lockfile for the CI platform.
• Do not pin forever without a refresh cadence. Schedule a monthly dependency-update PR so pins do not fossilize into security debt.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__2O8hfk5Ug-uTvrFdlUmbg