agent deployed the notebook to airflow without pinning dependencies, build failed

TL;DR

An unpinned notebook deployed to Airflow builds a different environment every time, so today's working DAG is tomorrow's broken one. Pin every dependency (pip freeze or a lockfile), build the Airflow image or env from the lockfile, and version the lockfile with the DAG. The agent's deploy step should refuse to ship without a lockfile. Reproducibility is the deploy requirement, not a nice-to-have.

agent deployed the notebook to airflow without pinning dependencies, build failed

Use this when

• An Airflow-deployed notebook fails with dependency errors that did not occur in dev.
• Rebuilds produce different environments.
• The deploy has no requirements file or lockfile.

Not for this skill when

• The DAG code itself is broken. That is a DAG bug.
• Pinned builds still fail. That is a genuine incompatibility.

Steps

1. Freeze the working dev environment: run pip freeze and save the output as requirements-lock.txt. Verify: the file pins every package.
2. Add the lockfile to the deploy (image build or env creation). Verify: the build consumes the lockfile.
3. Rebuild from scratch and run the notebook. Verify: the scheduled run matches dev.
4. Version the lockfile alongside the DAG. Verify: rollbacks restore the env too.
5. Gate the agent's deploy: no lockfile, no deploy. Verify: the gate fires on a test unpinned deploy.

Variant phrasings

airflow notebook dependencies not pinned

Short form. Lockfile or it did not happen.

notebook deploy airflow build failed

The symptom phrasing. Pin, then rebuild.

unpinned deps scheduled notebook

General phrasing. Reproducibility is the requirement.

Compatibility: Airflow 2.x, pip/conda. Lockfiles work with any builder.

Why it happens

Unpinned installs resolve to 'latest' at build time, and latest moves: a numpy 2 release, a removed API, a yanked wheel. The notebook that worked in dev was built against a specific resolution that no longer exists, so the Airflow build fails on a dependency combination nobody tested. A lockfile freezes the working resolution and makes every build identical.

Edge cases / pitfalls

• pip freeze captures everything, including dev-only tools. Curate or use a proper locker (pip-tools, poetry).
• The lockfile must match the python version too. A 3.11 lockfile on a 3.10 image breaks.
• Native dependencies (system libs) also need pinning in the image. The lockfile covers python only.
• Review lockfile diffs on update like code. A major bump deserves a test run.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3BO1C0MGrevGRl4MVijBg