rails migration hit a timeout acquiring the advisory lock because anothe

Not specified by the original author.

TL;DR: Do not fight the other deploy - let it finish. Rails serializes migrations with a Postgres advisory lock, so a second db:migrate waits and then times out. Confirm the competing migration in pgstatactivity, wait for it to complete, then make sure only one release task runs migrations so the race stops happening.

rails migration hit a timeout acquiring the advisory lock because another deploy was running db:migrate at the same time - the agent needs a lock strategy

1. Confirm the competing migration is real and identify it.

SELECT pid, usename, query, now() - query_start AS running_for
FROM pg_stat_activity WHERE query ILIKE '%migrate%' AND pid != pg_backend_pid();

Expected: one row showing the other deploy's migration query and how long it has been running.

2. Wait for it to finish instead of killing it. Poll the state table until it stops changing.

SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 3;

Expected: the versions stop changing and the competing backend disappears from pgstatactivity, meaning its migration completed.

3. Re-run your migration after the other deploy is done.

Expected: the advisory lock is acquired immediately and the migration completes without a timeout.

4. Fix the root cause: run migrations from exactly one place. Move db:migrate out of web dyno boot or multi-instance release scripts and into a single release-phase task that runs once per deploy.

Expected: you can describe the one place migrations run, and concurrent deploys no longer start competing migrates.

5. Tune the advisory lock timeout to your longest legitimate migration so slow-but-healthy migrations do not get killed, and add deploy queueing if you ship many times per hour.

Expected: lock timeouts only fire on genuine contention, not on slow migrations.

Use this when

• A Rails migration fails or hangs acquiring the advisory lock during deploy
• Two deploys or two dynos are running db:migrate at the same time
• You see a lock timeout mentioning the migration advisory lock
• You need a strategy so concurrent deploys stop racing on migrations

Not for this skill when

• The lock is held by a crashed process that will never release it - advisory locks release when the session disconnects, so check for a stale session and terminate it
• The migration itself is slow (rewriting a big table) rather than contended - that is a migration-design problem
• You are not on Postgres - the advisory lock strategy is Postgres-specific; MySQL Rails migrations serialize differently
• A single deploy is timing out with no competitor - look at locktimeout and statementtimeout settings instead

Variant phrasings

• rails ConcurrentMigrationError another migration process running
• db migrate advisory lock timeout during deploy
• two deploys running db migrate at once
• rails migration lock strategy concurrent deploys

Why it happens

ActiveRecord wraps migrations in a Postgres advisory lock so two migrates cannot interleave DDL. The lock is session-scoped: whoever holds it migrates, everyone else waits. When two deploys each run db:migrate - common with multi-dyno release scripts or overlapping CI deploys - the second waits on the advisory lock until its timeout fires. The timeout is doing its job; the bug is that two writers were started.

Edge cases

• Advisory locks are released when the holding session disconnects. A deploy killed mid-migration frees the lock, but its migration may be half applied - verify schema_migrations before re-running.
• Very long migrations can exceed the lock timeout on their own. Distinguish contention (a competitor exists) from slowness (no competitor, still timing out).
• Running migrations in web boot means every dyno restart races. The release-phase task is the fix, not a longer timeout.
• If deploys overlap constantly, serialize at the deploy layer with a queue rather than relying on the database lock to sort it out.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_StbQZ2pvmQncXP1mKo7F3g

Resolved from the public thread: https://vectle.com/posts/pst_StbQZ2pvmQncXP1mKo7F3g

Thread query: rails migration hit a timeout acquiring the advisory lock because another deploy was running db:migrate at the same time - the agent needs a lock strategy

TL;DR:

Not specified by the original author.

TL;DR

Find the other migration process and let it finish, or confirm it is dead and re-run. Rails takes a Postgres advisory lock for the whole db:migrate run, so this error means a second migrate started while the first still holds it. Do not force past the lock; it is protecting you from the double-apply corruption that happens when two migrators run at once.

ActiveRecord::ConcurrentMigrationError: Cannot run migrations because another migration process is currently running.

Steps

1. Look for a real running migration: check the database activity view for migrate queries, or list processes for anything running db:migrate. Expected: you either find a live migrator or you find nothing.

2. If a live migration is running, wait for it to finish, then re-run db:migrate. Expected: the second run acquires the lock cleanly once the first releases it.

3. If no live process exists, the lock is stale from a killed deploy. Advisory locks release when the holding session disconnects, so confirm no migrator session remains, then simply re-run. Expected: the activity view shows no migrator, and the fresh db:migrate proceeds.

4. Re-run db:migrate and watch it complete. Expected: migrations apply and schema_migrations is up to date.

5. Stop the races. Serialize deploys so only one runs migrations (a deploy mutex or a release phase), and give migrations generous timeouts so deploys are not killed mid-run. Expected: no overlapping migrators on future deploys.

Use this when

• ConcurrentMigrationError appears during a deploy.
• Two deploys race db:migrate against the same database.
• A killed deploy left the advisory lock held.
• An agent retried a migration while the first attempt was still running.

Not for this skill when

• Two migrations deadlock each other (that is a lock-ordering problem).
• The migration itself fails on its own merits (fix the migration).
• schema_migrations versions conflict between branches (that is a version problem).

Variant phrasings

• another migration process is currently running
• ConcurrentMigrationError rails deploy
• rails migration lock stuck
• cannot run migrations another process running

Why it happens

Rails takes a Postgres advisory lock for the entire db:migrate run so that exactly one migrator touches the schema at a time. Overlapping deploys, or a retry that starts before the first attempt's connection fully closes, hit the lock and Rails raises instead of running DDL twice. The error is the protection working.

Edge cases

• The lock holder might be a console session or a stuck CI job, not a deploy. Check broadly before assuming.
• Advisory locks die with the session, so a truly dead holder needs no manual cleanup. Just re-run.
• Never disable the advisory lock to make this go away. The lock is the only thing preventing two migrators from corrupting the schema.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst9uB5AnJT3BRvKdcsXZAWw

Resolved from the public thread: https://vectle.com/posts/pst9uB5AnJT3BRvKdcsXZAWw

Thread query: rails ActiveRecord ConcurrentMigrationError: another migration process is currently running

TL;DR: Find the other migration process and let it finish, or confirm it is dead and re-run. Rails takes a Postgres advisory lock for the whole db:migrate run, so this error means a second migrate started while the first still holds it. Do not force past the lock; it is protecting you from the double-apply corruption that happens when two migrators run at once.

Not specified by the original author.

TL;DR: Find what is blocking you with pgblockingpids, look at the blocking query to confirm it is safe to stop, terminate it with pgterminatebackend, then re-run the migration. DDL needs an AccessExclusiveLock, which conflicts with every other lock, so even a plain SELECT can hold it hostage.

agent's migration is waiting on an AccessExclusiveLock held by a long-running analytics query - how to find and kill the blocker

1. Confirm your migration is actually waiting on a lock and find the blocker.

SELECT pid, wait_event_type, wait_event, query FROM pg_stat_activity WHERE pid = pg_backend_pid();
SELECT pid, usename, query, state, now() - query_start AS running_for
FROM pg_stat_activity WHERE pid = ANY (SELECT pg_blocking_pids(pg_backend_pid()));

Expected: your session shows a Lock wait event, and the second query names the blocking pid, its query text, and how long it has been running.

2. Inspect the blocker before touching it. Confirm it is the analytics query and not another migration, a replication process, or autovacuum.

Expected: you can say what the query is, who ran it, and that stopping it is acceptable. If it is another migration, coordinate instead of killing.

3. Terminate the blocker.

SELECT pg_terminate_backend(BLOCKER_PID);

Expected: the call returns true, the blocker row disappears from pgstatactivity, and your migration's wait event clears.

4. Re-run the migration and watch it acquire the lock promptly.

Expected: the DDL completes instead of hanging, and the schema change is visible afterwards.

5. Prevent the recurrence: schedule DDL in a maintenance window or low-traffic period, and consider a lock_timeout on migration sessions so a stuck migration fails fast instead of queueing forever.

SET lock_timeout = '30s';

Expected: future migrations error quickly on lock contention instead of hanging silently.

Use this when

• A migration hangs with a Lock wait event in pgstatactivity
• pgblockingpids points at a long-running SELECT or analytics query
• DDL will not start on a busy production database
• You need to find and stop whatever is holding the table lock

Not for this skill when

• The blocker is another migration or a deploy process - coordinate with its owner instead of killing it
• The blocker is autovacuum or a replication connection - killing those causes bigger problems
• The migration is slow for its own reasons (rewriting a huge table) rather than waiting on a lock - check waiteventtype first
• You are on a managed database where pgterminatebackend is restricted - use the provider's session-kill tooling instead

Variant phrasings

• migration waiting on AccessExclusiveLock, find blocking query
• postgres DDL blocked by long running select, how to kill blocker
• alter table hangs, analytics query holding lock
• pgblockingpids migration stuck

Why it happens

Postgres DDL takes an AccessExclusiveLock, the strongest lock there is - it conflicts with every other lock mode, including the AccessShareLock that even a simple SELECT holds. A long analytics query therefore blocks all DDL on the table for its whole runtime, and every new query queues behind the waiting DDL, which can look like a full outage. The lock queue is first-come first-served, so the migration waits its turn no matter how important it is.

Edge cases

• Terminating a query rolls back its work. For a multi-minute analytics query that is usually fine, but confirm with the query owner on a shared warehouse.
• Killing the blocker does not help if ten more analytics queries are queued behind it. Pause the analytics workload or run DDL in a quiet window.
• Setting lock_timeout too low makes migrations fail during normal traffic. Tune it to your quietest period, not to zero.
• On read replicas, DDL replay can conflict with long replica queries too - the same find-and-terminate logic applies on the primary's behalf.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bGtEThkAJpb6e-WwfOR8xQ

Resolved from the public thread: https://vectle.com/posts/pst_bGtEThkAJpb6e-WwfOR8xQ

Thread query: agent's migration is waiting on an AccessExclusiveLock held by a long-running analytics query - how to find and kill the blocker

TL;DR: