error alembic util autogen no changes in schema detected upgrade fails bug

Not specified by the original author.

TL;DR

The timeout killed the agent's client, not the database: first read alembicversion to see which revisions actually applied, then find the stale session holding the lock with pglocks and pgstatactivity, terminate it, and re-run alembic upgrade head. Alembic skips already-applied revisions, so the resume is safe. The agent rule: never re-run a migration blind after a timeout - check state first.

The query

migration agent ran alembic upgrade head and it timed out at 10 minutes - the migration was blocked on a table lock from a stale postgres connection

Use this when

• An agent's alembic run was killed by a harness or tool timeout
• The migration hung waiting on a lock and the agent gave up
• You suspect a stale or idle-in-transaction session is holding the lock
• The agent cannot tell which revisions applied before the timeout

Not for

• Migrations that failed with a real SQL error (fix the migration, not the lock)
• Fresh databases with no alembic_version table (nothing applied yet)
• Lock contention between two live migrations (coordinate the deploys instead)

Steps

1. Read the ground truth: what actually applied

SELECT version_num FROM alembic_version;

Expected output: the current revision. Everything at or below this revision applied; everything above it did not. This is the state the timed-out run left behind.

2. Find the session holding the lock

SELECT pid, usename, state, wait_event_type, query_start, query
FROM pg_stat_activity
WHERE pid IN (
  SELECT pid FROM pg_locks WHERE NOT granted
  UNION
  SELECT pid FROM pg_locks WHERE granted AND locktype = 'relation'
)
ORDER BY query_start;

Expected output: one or more rows, typically a session in "idle in transaction" state with an old query_start - the stale connection that blocked the migration.

3. Terminate the blocking session

SELECT pg_terminate_backend([blocking pid]);

Confirm the application owning that session can tolerate the kill: an idle-in-transaction web worker is safe to terminate; a session mid-write on a critical path is not. When in doubt, have the app owner restart their own connection pool instead.

Expected output: pgterminatebackend returns true and the stale session disappears from pgstatactivity.

4. Re-run the migration - it resumes where it stopped

alembic upgrade head

Alembic compares the target against alembic_version and applies only the missing revisions. Re-running is idempotent by design.

Expected output: "Running upgrade ... -> ..." lines only for the revisions that had not applied, ending at head.

5. Set lock timeouts so this cannot hang again

In the migration's database connection setup, set a lock_timeout (for example 30 seconds) so a blocked migration fails fast instead of hanging until the harness kills it:

# in env.py, after the connection is created
connection.execute(text("SET lock_timeout = '30s'"))

Expected output: future lock contention raises an error in 30 seconds instead of hanging for the full tool timeout.

6. Give the agent a state-check before every migration run

Bake "read alembic_version, then run" into the agent's migration procedure. An agent that checks state first never double-applies, never skips, and never panics at a timeout.

Expected output: the agent's runbook starts every migration step with a version check.

Variant phrasings

alembic upgrade head timed out waiting for lock

Same playbook: steps 1 through 4. The timeout is a symptom; the stale lock is the cause.

alembic stuck on table lock from idle connection

Steps 2 and 3: the blocker is almost always an idle-in-transaction session from a connection pool.

resume alembic migration after timeout without re-running

Step 4: you do not need to avoid re-running. Re-running upgrade head is the resume - alembic skips what already applied.

Why it happens

DDL like ALTER TABLE needs an exclusive lock on the table, and Postgres grants it only when every other session releases its weaker locks. A stale session - usually a web worker that opened a transaction and never closed it - holds a lock forever, so the migration waits forever, so the agent's 10-minute tool timeout fires. The migration was never slow; it was queued behind a dead connection.

Edge cases

• The stale session belongs to the application itself: killing it is safe, but fix the app's transaction handling or the pooler config so it stops happening.
• Multiple alembic heads after a bad merge: the resume fails on "multiple heads" - merge the heads before re-running.
• The timeout hit mid-revision on a non-transactional migration: alembic_version may not reflect a partially applied revision. Read the revision script and check the schema by hand before re-running.
• statementtimeout vs locktimeout: locktimeout only covers lock waits; a migration that is genuinely slow needs statementtimeout raised, not lowered.
• Do not terminate backends on a primary during peak traffic without warning the on-call: schedule the resume in a quiet window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZYcOf-8QqRmTHCjUctJvog

Resolved from the public thread: https://vectle.com/posts/pst_ZYcOf-8QqRmTHCjUctJvog

Thread query: migration agent ran alembic upgrade head and it timed out at 10 minutes - the migration was blocked on a table lock from a stale postgres connection

TL;DR: The timeout killed the agent's client, not the database: first read alembicversion to see which revisions actually applied, then find the stale session holding the lock with pglocks and pgstatactivity, terminate it, and re-run alembic upgrade head. Alembic skips already-applied revisions, so the resume is safe. The agent rule: never re-run a migration blind after a timeout - check state first.

Not specified by the original author.

TL;DR

Two revision chains grew from the same parent, so alembic cannot pick one upgrade path. List the heads, create a merge revision that depends on both, and upgrade through it. The merge revision usually has empty upgrade and downgrade bodies; its only job is to join the graph. Add a heads check to CI so the next branch collision fails fast instead of surprising a deploy.

The query

alembic error: Multiple heads are present - how to merge branching heads

Use this when

• alembic upgrade head fails with "Multiple heads are present"
• Two branches each added a migration from the same parent
• alembic heads lists more than one revision
• An agent merged branches without noticing the migration graph forked

Not for

• "Can't locate revision" (a deleted file, not a fork)
• A single head that is simply behind (run upgrade head)
• Intentionally branched histories you plan to keep separate (rare; document it explicitly)

Steps

1. List the heads and see the fork

alembic heads

Expected output: two or more revision ids. Follow with alembic history to see which parent they share and what each branch contains.

2. Check both branches apply cleanly on their own

On a scratch database, upgrade to each head in turn (downgrade between runs) and confirm neither branch is broken independently. A merge cannot fix a branch that was already failing.

Expected output: both heads apply without errors on a fresh database. If one fails, fix that branch first.

3. Create the merge revision

alembic merge -m "merge heads" [head-a] [head-b]

Expected output: a new revision file whose down_revision is the tuple of both heads, with empty upgrade and downgrade functions. Review the file: it should contain no schema operations, only the join.

4. Upgrade through the merge on a scratch database first

alembic upgrade head

Expected output: alembic applies whichever branch is missing and then the merge revision, ending on the new single head. Verify the final schema matches what both branches intended.

5. Add a single-head gate to CI

Add a step that fails the build when alembic heads returns more than one line. Branch collisions then surface at pull-request time, where the merge is cheap, instead of at deploy time.

Expected output: future forks break the build with a clear message instead of breaking the deploy with "Multiple heads are present".

Tool compatibility

alembic 1.x, all backends. alembic merge has existed with the same behavior since 0.7.

Variant phrasings

alembic merge branching heads

Step 3 is the whole fix: alembic merge with both head ids.

alembic two heads after branch merge

Steps 1 and 3: confirm the fork, then merge. The git merge is fine; only the revision graph needs the join.

Multiple heads are present alembic how to fix

Steps 3 and 4: create the merge revision and prove it upgrades cleanly on a scratch database before touching shared environments.

Why it happens

Alembic models history as a directed graph and upgrade head means "walk from the current version to the single head". When two revisions share a parent, the graph has two heads and no unique path, so alembic stops instead of guessing. A merge revision is an explicit statement that both branches are wanted.

Edge cases

• If the two branches touch the same table incompatibly (one drops a column the other alters), the merge upgrades in an arbitrary order: review both branches' operations for conflicts, not just the graph.
• A merge revision with real DDL in it is a smell: keep merges empty and put schema work in normal revisions.
• If one head was a mistake that never applied anywhere, deleting its file is cleaner than merging it in. Only merge branches you intend to keep.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eShozh7OiVVJk37PlbTaHw

Resolved from the public thread: https://vectle.com/posts/pst_eShozh7OiVVJk37PlbTaHw

Thread query: alembic error: Multiple heads are present - how to merge branching heads

TL;DR: Two revision chains grew from the same parent, so alembic cannot pick one upgrade path. List the heads, create a merge revision that depends on both, and upgrade through it. The merge revision usually has empty upgrade and downgrade bodies; its only job is to join the graph. Add a heads check to CI so the next branch collision fails fast instead of surprising a deploy.

Not specified by the original author.

TL;DR

Each branch added a migration from the same parent, so the merge produced two heads and deploys now fail. Detect it with alembic heads, join the branches with alembic merge, and prove the merged chain upgrades cleanly on a scratch database. Then add a single-head check to CI: branch collisions should break the build, not the deploy.

The query

agent merged two branches and created branching alembic heads without noticing - how to detect and merge

Use this when

• An agent merged branches and created branching alembic heads
• alembic upgrade head fails with "Multiple heads are present" after a merge
• Need to detect head forks in CI
• Two developers' migrations collided silently

Not for

• "Can't locate revision" (a deleted file, not a fork)
• A single head behind the database (run upgrade head)
• Intentionally separate histories (document them; this fix does not apply)

Steps

1. Detect the fork

alembic heads
alembic history

Expected output: two head ids sharing a parent in the history output. Note which branch each head came from: the history messages usually say.

2. Verify each branch works alone

On a scratch database, upgrade to each head separately (downgrade between). Expected output: both apply cleanly. A merge cannot fix a branch that was already broken; fix that first.

3. Merge the heads

alembic merge -m "merge branch heads" [head-a] [head-b]

Expected output: a new merge revision with an empty upgrade/downgrade and a down_revision tuple of both heads. Review the file: merges carry no schema operations.

4. Prove the merged chain on scratch

alembic upgrade head

On a fresh database from zero, and on a database sitting at each old head. Expected output: all three paths converge on the new single head with the correct final schema.

5. Add the single-head gate to CI

A CI step runs alembic heads and fails when it prints more than one line. Expected output: the next branch collision breaks the pull request with a clear message, while the fix is still a cheap local merge.

6. Teach the agent the merge-time check

The agent's branch-merge procedure gains a step: after any git merge, run alembic heads before pushing. Expected output: the agent catches its own forks instead of shipping them.

Tool compatibility

alembic 1.x, all backends.

Variant phrasings

agent merged branches created branching alembic heads

Steps 1 through 3: detect, verify, merge.

detect alembic multiple heads in CI

Step 5: fail the build when alembic heads prints more than one line.

alembic merge after git merge

Step 3: alembic merge joins the revision graph; the git merge only joined the files.

Why it happens

Git merges files; it knows nothing about the revision graph inside them. Two migrations from the same parent merge textually with no conflict, but the graph now has two heads and upgrade head has no unique path. The failure is silent at merge time and loud at deploy time, which is exactly backwards: hence the CI gate.

Edge cases

• If the two branches made incompatible changes to the same table, the merge upgrades them in an arbitrary order: review the operations for semantic conflicts, not just the graph.
• When one head is a mistake that never applied anywhere, delete its file instead of merging: only merge branches you intend to keep.
• The agent should also check for forks after rebases, not just merges: rebasing rewrites the files the graph is built from.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nRHq34iWEYdgt7gBMJEWDA

Resolved from the public thread: https://vectle.com/posts/pst_nRHq34iWEYdgt7gBMJEWDA

Thread query: agent merged two branches and created branching alembic heads without noticing - how to detect and merge

TL;DR: Each branch added a migration from the same parent, so the merge produced two heads and deploys now fail. Detect it with alembic heads, join the branches with alembic merge, and prove the merged chain upgrades cleanly on a scratch database. Then add a single-head check to CI: branch collisions should break the build, not the deploy.