airflow error: airflow db migrate failed with alembic revision conflict

TL;DR

Airflow manages its metadata database with its own alembic chain, and jumping versions (or mixing airflow versions against one database) forks or gaps that chain. Check that the installed airflow version matches the database's migration state, upgrade through intermediate versions when you skipped, then run airflow db migrate again. Never hand-edit airflow's alembic history.

The query

airflow error: airflow db migrate failed with alembic revision conflict

Use this when

• airflow db migrate fails with an alembic revision conflict
• Airflow upgrade left the metadata database unmigratable
• Multiple heads or missing revisions in airflow's own migrations
• An agent upgraded airflow and the scheduler will not start

Not for

• DAG import or task failures (those are user code, not the metadata db)
• Alembic conflicts in your own application's migrations (separate chain, separate fix)
• A fresh airflow install (just run airflow db migrate from zero)

Steps

1. Check the version alignment

airflow version

And inspect the metadata database's alembic state:

SELECT version_num FROM alembic_version;

Expected output: the installed airflow version and the database's recorded revision. A mismatch (db newer than the installed code, or far older) is the conflict's cause.

2. Find the supported upgrade path

Read the airflow release notes for the versions between the database's state and the installed version. Airflow sometimes requires stepping through an intermediate release because a migration was rewritten.

Expected output: the sequence of versions to install, e.g. 2.7.x before 2.9.x. Skipping is what forked the chain.

3. Step through the intermediate versions

Install each intermediate version in turn and run its migration before moving on:

airflow db migrate

Expected output: each step's migration completes cleanly. Slow but safe: every intermediate migration was tested by the project in exactly this order.

4. Re-run the final migration

With the correct airflow version installed:

airflow db check

then

airflow db migrate

Expected output: db check reports the database is ready, and migrate completes with no revision conflict.

5. Pin airflow and database together going forward

The agent's upgrade runbook becomes: check the upgrade path, step through intermediates, migrate at each step, and never run two different airflow versions against the same metadata database.

Expected output: the next upgrade follows the path instead of jumping it.

Tool compatibility

apache airflow 2.7.x through 2.10.x. airflow db migrate and db check are stable across these.

Variant phrasings

airflow db migrate alembic revision conflict

Steps 1 through 3: align versions, step through intermediates.

airflow multiple alembic heads metadata db

Step 3: the intermediate versions contain the merge or repair migrations the project wrote for exactly this situation.

airflow scheduler will not start after upgrade db migrate failed

Step 4: the scheduler needs the metadata migration to complete; fix the chain first, then start the scheduler.

Why it happens

Airflow ships its own alembic revisions with the package, and the metadata database records which revision it is at. Installing a far-newer airflow changes the available revision graph underneath a database stamped with an old revision; if the project rewrote history between the two, the graphs no longer connect. The conflict is version skew between code and database, not corruption.

Edge cases

• Downgrading airflow to match the database is sometimes the fastest recovery: install the version the database expects, migrate, then step forward properly.
• Two schedulers on different airflow versions sharing one metadata database will fork the state: keep every component on the same version.
• Back up the metadata database before the stepping procedure: it holds DAG run history you cannot reconstruct.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mJ8k4W8AgTk4jHqzFo-WyA