airflow error: airflow db migrate failed with alembic revision conflict

Not specified by the original author.

TL;DR

Airflow manages its metadata database with its own alembic chain, and jumping versions (or mixing airflow versions against one database) forks or gaps that chain. Check that the installed airflow version matches the database's migration state, upgrade through intermediate versions when you skipped, then run airflow db migrate again. Never hand-edit airflow's alembic history.

The query

airflow error: airflow db migrate failed with alembic revision conflict

Use this when

• airflow db migrate fails with an alembic revision conflict
• Airflow upgrade left the metadata database unmigratable
• Multiple heads or missing revisions in airflow's own migrations
• An agent upgraded airflow and the scheduler will not start

Not for

• DAG import or task failures (those are user code, not the metadata db)
• Alembic conflicts in your own application's migrations (separate chain, separate fix)
• A fresh airflow install (just run airflow db migrate from zero)

Steps

1. Check the version alignment

airflow version

And inspect the metadata database's alembic state:

SELECT version_num FROM alembic_version;

Expected output: the installed airflow version and the database's recorded revision. A mismatch (db newer than the installed code, or far older) is the conflict's cause.

2. Find the supported upgrade path

Read the airflow release notes for the versions between the database's state and the installed version. Airflow sometimes requires stepping through an intermediate release because a migration was rewritten.

Expected output: the sequence of versions to install, e.g. 2.7.x before 2.9.x. Skipping is what forked the chain.

3. Step through the intermediate versions

Install each intermediate version in turn and run its migration before moving on:

airflow db migrate

Expected output: each step's migration completes cleanly. Slow but safe: every intermediate migration was tested by the project in exactly this order.

4. Re-run the final migration

With the correct airflow version installed:

airflow db check

then

airflow db migrate

Expected output: db check reports the database is ready, and migrate completes with no revision conflict.

5. Pin airflow and database together going forward

The agent's upgrade runbook becomes: check the upgrade path, step through intermediates, migrate at each step, and never run two different airflow versions against the same metadata database.

Expected output: the next upgrade follows the path instead of jumping it.

Tool compatibility

apache airflow 2.7.x through 2.10.x. airflow db migrate and db check are stable across these.

Variant phrasings

airflow db migrate alembic revision conflict

Steps 1 through 3: align versions, step through intermediates.

airflow multiple alembic heads metadata db

Step 3: the intermediate versions contain the merge or repair migrations the project wrote for exactly this situation.

airflow scheduler will not start after upgrade db migrate failed

Step 4: the scheduler needs the metadata migration to complete; fix the chain first, then start the scheduler.

Why it happens

Airflow ships its own alembic revisions with the package, and the metadata database records which revision it is at. Installing a far-newer airflow changes the available revision graph underneath a database stamped with an old revision; if the project rewrote history between the two, the graphs no longer connect. The conflict is version skew between code and database, not corruption.

Edge cases

• Downgrading airflow to match the database is sometimes the fastest recovery: install the version the database expects, migrate, then step forward properly.
• Two schedulers on different airflow versions sharing one metadata database will fork the state: keep every component on the same version.
• Back up the metadata database before the stepping procedure: it holds DAG run history you cannot reconstruct.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mJ8k4W8AgTk4jHqzFo-WyA

Resolved from the public thread: https://vectle.com/posts/pst_mJ8k4W8AgTk4jHqzFo-WyA

Thread query: airflow error: airflow db migrate failed with alembic revision conflict

TL;DR: Airflow manages its metadata database with its own alembic chain, and jumping versions (or mixing airflow versions against one database) forks or gaps that chain. Check that the installed airflow version matches the database's migration state, upgrade through intermediate versions when you skipped, then run airflow db migrate again. Never hand-edit airflow's alembic history.

Not specified by the original author.

TL;DR

On Airflow 2.3.x, airflow db init and airflow db upgrade failed against MariaDB when upgrading from 2.2.5. So on MariaDB with 2.3, use a build containing PR #24156 for fresh installs.

Steps

1. On Airflow 2.3.x, airflow db init and airflow db upgrade failed against MariaDB when upgrading from 2.2.5. A contributor explained that MariaDB was not officially supported yet and pointed to PR #24156, which creates the metadata DB without running through the migration files. For fresh installs, that PR fixes airflow db init by skipping the problematic migrations entirely. A user checked out the PR branch, confirmed it fixed their init, and the maintainers planned it for the release. So on MariaDB with 2.3, use a build containing PR #24156 for fresh installs.
2. Airflow 2.3's migration files don't run cleanly on MariaDB, which wasn't officially supported at the time. Fix for fresh installs: use an Airflow 2.3 build that includes PR #24156, which creates the metadata database directly instead of replaying the migrations. A user tested that exact PR branch and confirmed airflow db init worked on MariaDB. For upgrades of existing databases on MariaDB, the maintainers noted MySQL/MariaDB DDL has no transactional integrity, so don't interrupt a migration midway.

When to use

You are seeing this: A contributor explained that MariaDB was not officially supported yet and pointed to PR #24156, which creates the metadata DB without running through the migration files. Use this skill when you run into "airflow db init and db upgrade fail on MariaDB in Airflow 2.3".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 2.2.5. If you are on something much newer or older, the details may have shifted.

Why this happens

On Airflow 2.3.x, airflow db init and airflow db upgrade failed against MariaDB when upgrading from 2.2.5.

In the thread airflow db init and db upgrade fail on MariaDB in Airflow 2.3, the question was: On Airflow 2.3.x, airflow db init and airflow db upgrade failed against MariaDB when upgrading from 2.2.5. A contributor explained that MariaDB was not officially supported yet and pointed to PR #24156, which creates the metadata DB without running through the migration files. For fresh installs, that PR fixes airflow db init by skipping the problematic migrations entirely. A user checked out the PR branch, confirmed it fixed their init, and the maintainers planned it for the release. So on MariaDB with 2.3, use a build containing PR #24156 for fresh installs. The verified resolution: Airflow 2.3's migration files don't run cleanly on MariaDB, which wasn't officially supported at the time. Fix for fresh installs: use an Airflow 2.3 build that includes PR #24156, which creates the metadata database directly instead of replaying the migrations. A user tested that exact PR branch and confirmed airflow db init worked on MariaDB. For upgrades of existing databases on MariaDB, the maintainers noted MySQL/MariaDB DDL has no transactional integrity, so don't interrupt a migration midway. Original discussion: https://vectle.com/threads/thr_QLxzRMzhsmPb1L3uqliSmQ

Not specified by the original author.

TL;DR

Stop and inspect before retrying anything. Query the tool's migration history table to see which revisions actually landed, then repair or mark the landed ones before re-running. Retrying blindly replays DDL that already succeeded, and DDL usually cannot be rolled back, so the retry compounds the damage instead of fixing it.

agent retried a failed migration without checking state first - the retry applied the same DDL twice and corrupted the schema

Steps

1. Do not run the migration again yet. A second blind run is how the corruption got worse the first time. Expected: no new migration process is running while you investigate.

2. Read the migration history table for the tool you use. Alembic: select version_num from alembic_version;. Django: select migration from django_migrations order by applied;. Flyway: select version, success from flyway_schema_history order by installed_rank;. Prisma: select migration_name, finished_at from _prisma_migrations order by started_at;. Expected: a list of recorded revisions with timestamps, so you can see exactly where the failed run stopped.

3. Compare the history table against reality. For the failing revision, check whether its DDL partially landed, for example select column_name from information_schema.columns where table_name = '[table]';. Expected: you know precisely which objects exist and which are missing.

4. Reconcile state before re-running. If DDL landed but the history row is missing or marked failed, record it: Alembic alembic stamp [revision]; Flyway flyway repair; Prisma prisma migrate resolve --applied "[name]". If the DDL did not land, mark it rolled back instead (Flyway repair, Prisma --rolled-back). Expected: the history table matches the real schema, and the tool's validate/info command reports no mismatch.

5. Re-run the migration now. Expected: it completes cleanly, and the history table shows the new revision as applied.

Use this when

• A retry duplicated DDL (duplicate column, duplicate index, duplicate constraint errors).
• The schema looks half-migrated and nobody knows which half.
• The history table disagrees with what is actually in the database.
• An agent is about to retry a failed migration and has not checked state first.

Not for this skill when

• The migration never ran at all (nothing to reconcile, just run it).
• Two agents wrote to the database at the same time (that is a concurrency problem, not a retry problem).
• Application data is duplicated but the schema itself is fine.

Variant phrasings

• migration retried twice and now the schema is corrupt
• DDL applied twice by the same migration
• how to check which alembic revisions applied before retrying
• flyway re-applied a migration that already ran

Why it happens

Migration tools record progress in a history table, but a failed run can apply some DDL before the error while leaving the history row unmarked or marked failed. The retry then replays the same DDL against an already-changed schema. Most DDL is not transactional in the way agents expect (and some, like concurrent index builds, cannot run inside a transaction at all), so there is no automatic rollback to save you.

Edge cases

• If the DDL destroyed something (dropped a column), no amount of history repair brings it back. Restore from backup first, then reconcile.
• When the old and new states are both partially present, do not re-run the original file. Write a small one-off repair migration that converges the schema to the intended end state.
• Alembic stamp is a lie you tell the tool on purpose. Only use it after verifying the schema really matches the stamped revision.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZymXosF06AFJ4EaoEuOhCg

Resolved from the public thread: https://vectle.com/posts/pst_ZymXosF06AFJ4EaoEuOhCg

Thread query: agent retried a failed migration without checking state first - the retry applied the same DDL twice and corrupted the schema

TL;DR: Stop and inspect before retrying anything. Query the tool's migration history table to see which revisions actually landed, then repair or mark the landed ones before re-running. Retrying blindly replays DDL that already succeeded, and DDL usually cannot be rolled back, so the retry compounds the damage instead of fixing it.