## TL;DR
An unpinned `pip install ipykernel` in CI floats to the newest release, and a new ipykernel (or its new traitlets, jupyter_client) can break notebook execution that worked yesterday. Pin `ipykernel`, `nbclient`, `nbformat`, and `jupyter_client` in the CI requirements and rebuild from a lockfile. Then re-run the notebook in CI to confirm the pinned stack executes it cleanly.

```text
agent ran the notebook in CI without pinning ipykernel: version drift crash
```

## Use this when
- A notebook that passed CI last week fails now with no code changes, after an ipykernel or dependency upgrade.
- The CI log shows a fresh `pip install ipykernel` pulling a new major or minor version.
- An agent generates the CI environment without a lockfile.

## Not for this skill when
- The failure is a labextension vs JupyterLab version conflict. That is the Lab 4 extension problem, a different fix.
- The kernel starts but user packages are missing. That is an incomplete requirements file, not version drift.

## Steps
1. Find the drift: compare the CI log's installed versions (`pip freeze` output in the log) against the last green run. Expected: you can name the package and version that changed.
2. Pin the execution stack in the CI requirements: `ipykernel`, `nbclient`, `nbformat`, `jupyter_client`, `traitlets`, each with `==` pins. Expected: the requirements file names exact versions.
3. Better, generate a lockfile from a known-good environment and have CI install from it. Expected: CI installs byte-identical dependencies on every run.
4. Clear the CI cache for the Python environment so the next run builds from the pins, not from a stale cached layer. Expected: the build log shows the pinned versions installing.
5. Re-run the notebook execution step in CI. Expected: the notebook executes cleanly with the pinned stack.
6. Add a CI step that fails when the lockfile is out of sync with the requirements, so the next drift is caught at PR time. Expected: version changes become deliberate PRs, not silent breakage.

## Tool compatibility
ipykernel 6.x; nbclient 0.10.x; nbformat 5.x; jupyter_client 8.x; pip or conda CI environments.

## Variant phrasings
### ipykernel version drift broke CI notebook
The drift phrasing. Pin the execution stack and install from a lockfile.

### notebook CI failed after ipykernel upgrade
The upgrade phrasing. The fix is pins, not chasing the new version's changes.

### agent CI notebook unpinned dependencies crash
The agent phrasing. Agents that write CI configs must emit pins, not bare package names.

## Why it happens
The notebook execution stack is several tightly-coupled packages, and ipykernel's major releases regularly change message-protocol details, asyncio handling, and dependency floors. CI that installs unpinned packages re-resolves the world on every run, so a passing pipeline can break overnight with zero code changes, and the failure looks like a notebook bug. Pinning converts dependency changes from accidents into reviewed decisions. A lockfile goes further by freezing the transitive closure, which is where the real surprises (a new `traitlets` or `jupyter_client`) usually come from.

## Edge cases / pitfalls
- Pinning only `ipykernel` still leaves `jupyter_client` and `traitlets` floating. Pin the whole execution stack or use a lockfile.
- Stale Docker layer caches can keep installing the old unpinned set even after you add pins. Bust the cache on requirements changes.
- A lockfile generated on one platform may not install on another (macOS vs Linux wheels). Generate the lockfile for the CI platform.
- Do not pin forever without a refresh cadence. Schedule a monthly dependency-update PR so pins do not fossilize into security debt.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst__2O8hfk5Ug-uTvrFdlUmbg