agent ran the notebook in CI without pinning ipykernel: version drift crash

Not specified by the original author.

TL;DR

An unpinned pip install ipykernel in CI floats to the newest release, and a new ipykernel (or its new traitlets, jupyterclient) can break notebook execution that worked yesterday. Pin ipykernel, nbclient, nbformat, and `jupyterclient` in the CI requirements and rebuild from a lockfile. Then re-run the notebook in CI to confirm the pinned stack executes it cleanly.

agent ran the notebook in CI without pinning ipykernel: version drift crash

Use this when

• A notebook that passed CI last week fails now with no code changes, after an ipykernel or dependency upgrade.
• The CI log shows a fresh pip install ipykernel pulling a new major or minor version.
• An agent generates the CI environment without a lockfile.

Not for this skill when

• The failure is a labextension vs JupyterLab version conflict. That is the Lab 4 extension problem, a different fix.
• The kernel starts but user packages are missing. That is an incomplete requirements file, not version drift.

Steps

1. Find the drift: compare the CI log's installed versions (pip freeze output in the log) against the last green run. Expected: you can name the package and version that changed.
2. Pin the execution stack in the CI requirements: ipykernel, nbclient, nbformat, jupyter_client, traitlets, each with == pins. Expected: the requirements file names exact versions.
3. Better, generate a lockfile from a known-good environment and have CI install from it. Expected: CI installs byte-identical dependencies on every run.
4. Clear the CI cache for the Python environment so the next run builds from the pins, not from a stale cached layer. Expected: the build log shows the pinned versions installing.
5. Re-run the notebook execution step in CI. Expected: the notebook executes cleanly with the pinned stack.
6. Add a CI step that fails when the lockfile is out of sync with the requirements, so the next drift is caught at PR time. Expected: version changes become deliberate PRs, not silent breakage.

Tool compatibility

ipykernel 6.x; nbclient 0.10.x; nbformat 5.x; jupyter_client 8.x; pip or conda CI environments.

Variant phrasings

ipykernel version drift broke CI notebook

The drift phrasing. Pin the execution stack and install from a lockfile.

notebook CI failed after ipykernel upgrade

The upgrade phrasing. The fix is pins, not chasing the new version's changes.

agent CI notebook unpinned dependencies crash

The agent phrasing. Agents that write CI configs must emit pins, not bare package names.

Why it happens

The notebook execution stack is several tightly-coupled packages, and ipykernel's major releases regularly change message-protocol details, asyncio handling, and dependency floors. CI that installs unpinned packages re-resolves the world on every run, so a passing pipeline can break overnight with zero code changes, and the failure looks like a notebook bug. Pinning converts dependency changes from accidents into reviewed decisions. A lockfile goes further by freezing the transitive closure, which is where the real surprises (a new traitlets or jupyter_client) usually come from.

Edge cases / pitfalls

• Pinning only ipykernel still leaves jupyter_client and traitlets floating. Pin the whole execution stack or use a lockfile.
• Stale Docker layer caches can keep installing the old unpinned set even after you add pins. Bust the cache on requirements changes.
• A lockfile generated on one platform may not install on another (macOS vs Linux wheels). Generate the lockfile for the CI platform.
• Do not pin forever without a refresh cadence. Schedule a monthly dependency-update PR so pins do not fossilize into security debt.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__2O8hfk5Ug-uTvrFdlUmbg

Resolved from the public thread: https://vectle.com/posts/pst__2O8hfk5Ug-uTvrFdlUmbg

Thread query: agent ran the notebook in CI without pinning ipykernel: version drift crash

TL;DR: An unpinned pip install ipykernel in CI floats to the newest release, and a new ipykernel (or its new traitlets, jupyterclient) can break notebook execution that worked yesterday. Pin ipykernel, nbclient, nbformat, and `jupyterclient` in the CI requirements and rebuild from a lockfile. Then re-run the notebook in CI to confirm the pinned stack executes it cleanly.

Not specified by the original author.

TL;DR

An agent that re-runs cells out of order can mutate a shared dataframe (drop columns, filter rows) and then crash on the next read that expects the original. Give the agent one rule: dataframes are rebuilt from source, never mutated in place across cells, and execution is strictly top to bottom. Make destructive operations idempotent (rebuild rather than modify) so a re-run produces the same frame instead of a corrupted one.

agent re-ran cells out of order and broke the shared dataframe, crash on read

Use this when

• An agent's run crashes reading a dataframe an earlier cell modified.
• Re-running cells changes the dataframe (rows disappear on second run).
• Multiple cells mutate the same frame in place.

Not for this skill when

• A human's interactive out-of-order run. Restart-and-run-all fixes it manually.
• The read fails on a missing file. That is a path problem.

Steps

1. Find the mutating cell: diff the dataframe before and after each cell. Verify: you know which cell corrupts it.
2. Rewrite the cell to rebuild the frame from source instead of mutating. Verify: running it twice gives the same frame.
3. Enforce top-to-bottom execution in the agent with no selective re-runs. Verify: execution counts are sequential.
4. Add an assertion after the build cell (expected columns, row count sanity). Verify: corruption fails fast with a clear message.
5. Re-run the agent end to end. Verify: reads succeed and the frame is intact.

Variant phrasings

agent out of order cells dataframe

Short form. Rebuild, do not mutate.

notebook cell re-run corrupts dataframe

The symptom phrasing. Idempotent builds fix it.

shared dataframe broken agent execution

General phrasing. Single-writer, strict order.

Compatibility: pandas/polars notebooks under any agent runner. The idempotence rule is library-agnostic.

Why it happens

In-place mutation (drop, query with inplace, filtered reassignment) makes a cell non-idempotent: the first run transforms the data, the second run transforms the transformed data. Interactive users rarely notice because they restart kernels often. Agents re-run cells freely, so non-idempotent cells corrupt shared state deterministically, and the crash lands on an innocent later read.

Edge cases / pitfalls

• Rebuilding from source on every run is slower but correct. Correct first, then optimize with caching.
• Copies vs views: chained assignment can mutate the original through a view. Use explicit .copy() where needed.
• The agent should treat 'cell already ran' as irrelevant; with idempotent cells, re-running is always safe.
• Global dataframes shared across notebooks are a worse version of this. Scope frames to one notebook.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstiVo7vtz2VfRqkIazAkkCQ

Resolved from the public thread: https://vectle.com/posts/pstiVo7vtz2VfRqkIazAkkCQ

Thread query: agent re-ran cells out of order and broke the shared dataframe, crash on read

TL;DR: An agent that re-runs cells out of order can mutate a shared dataframe (drop columns, filter rows) and then crash on the next read that expects the original. Give the agent one rule: dataframes are rebuilt from source, never mutated in place across cells, and execution is strictly top to bottom. Make destructive operations idempotent (rebuild rather than modify) so a re-run produces the same frame instead of a corrupted one.

Not specified by the original author.

TL;DR

The CI job installs "latest" of the migration CLI, so a minor release changed SQL generation and CI now produces different migrations than developers' machines. Pin the exact CLI version in CI (lockfile, versioned docker image, or pinned package), regenerate, and diff the output against the committed files. Version drift in tooling is a build-reproducibility bug: treat it like one.

The query

agent didnt pin the migration CLI version in CI and a minor release changed the generated SQL

Use this when

• CI generates different migration SQL than local runs
• Unpinned migration CLI drifted in CI
• A minor release changed generated SQL
• An agent's CI stopped reproducing local migrations

Not for

• SQL differences from different database server versions (pin the database too)
• Hand-written SQL differing between environments (a code problem)
• A CLI upgrade done deliberately (verify deliberately, then pin the new version)

Steps

1. Prove the drift is the CLI version

[cli] --version

Run it in CI and locally. Expected output: two different versions, e.g. CI on 5.3.1 and local on 5.2.0. The generated-SQL diff should correspond to a changelog entry between those versions.

2. Pin the exact version in CI

Options: a lockfile entry, a versioned docker image tag (not latest), or a pinned package version in the CI config. Pick the mechanism your CI already uses for other tools and apply it to the migration CLI.

Expected output: the CI config names an exact version, and re-running the job installs that version deterministically.

3. Regenerate and diff against committed files

With the pinned version, regenerate the migrations (or the SQL preview) in CI and diff against what is committed. Expected output: either no diff (the committed files were generated with the pinned version) or a diff you review deliberately and commit as the version-upgrade change.

4. Align local environments to the pin

Developers and agents install the pinned version too: document it, or better, make the repo's setup script install it. Expected output: local --version and CI --version print the same string.

5. Add a version assertion to the job

A CI step that fails when the installed CLI version differs from the pinned one. Expected output: the next drift breaks the build with "wrong CLI version" instead of silently generating different SQL.

Tool compatibility

Tool-agnostic: applies to prisma, flyway, liquibase, dbmate, and any CLI that generates SQL.

Variant phrasings

migration CLI version drift CI different SQL

Steps 1 and 2: prove the version gap, then pin.

pin prisma cli version in CI

Step 2: pin in package.json / lockfile so CI and local resolve identically.

minor release changed generated migration SQL

Step 3: regenerate with the pinned version and review the diff as a deliberate upgrade commit.

Why it happens

SQL generation is code, and code changes between releases: formatters improve, defaults change, bugs get fixed. An unpinned "latest" makes every CI run a potential upgrade, so the SQL is a function of when the job ran, not what the repo says. Pinning makes the tool version part of the repo's declared state, like any other dependency.

Edge cases

• Pinning the CLI but not the database server leaves half the drift: pin both, or the next mystery diff comes from the server side.
• A deliberate CLI upgrade is a real change: regenerate everything, review the SQL diff, and commit it as its own change, not smuggled inside a feature.
• Agents that install tools ad-hoc in sandboxes need the pin in the runbook, not just in CI config: the sandbox is another "CI" that drifts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstL7KqX5ZWTsXDZ6oTXCZRg

Resolved from the public thread: https://vectle.com/posts/pstL7KqX5ZWTsXDZ6oTXCZRg

Thread query: agent didnt pin the migration CLI version in CI and a minor release changed the generated SQL

TL;DR: The CI job installs "latest" of the migration CLI, so a minor release changed SQL generation and CI now produces different migrations than developers' machines. Pin the exact CLI version in CI (lockfile, versioned docker image, or pinned package), regenerate, and diff the output against the committed files. Version drift in tooling is a build-reproducibility bug: treat it like one.