agent's health check queried a table the migration had locked and

Not specified by the original author.

TL;DR

The health check ran a real query against a table the migration had locked, timed out, and the deploy was marked failed even though the app was fine. Point health checks at something the migration never locks (a trivial SELECT 1, a dedicated status table, or the migration-independent endpoint), add retries with backoff, and re-run the deploy. Health checks must measure app liveness, not schema availability.

The query

agent's health check queried a table the migration had locked and the deploy was marked failed incorrectly

Use this when

• Health check failed during a migration and marked the deploy failed
• Health check queries a table the migration locks
• Spurious deploy failures during schema changes
• An agent's deploy keeps "failing" only while migrations run

Not for

• Health checks failing because the app is actually down (fix the app)
• Migration failures themselves (fix the migration)
• Load-balancer health checks with no database involvement (a different config)

Steps

1. Confirm the failure was the lock, not the app

Correlate timestamps: the health check failures should coincide exactly with the migration's lock window, and the app should have been serving fine otherwise. Check the migration logs for the locked table and the health check logs for the timed-out query.

Expected output: the two timelines overlap perfectly, and no app errors exist outside the window.

2. Point the health check away from migrated tables

Change the health check to a query the migration never touches:

SELECT 1;

Or a lightweight endpoint that does not hit the database at all. Expected output: the health check passes regardless of which tables are locked, because it no longer depends on them.

3. Add retries with backoff to the check

Even a cheap check can hit a busy moment: configure the health checker to retry a few times with backoff before marking the deploy failed. Expected output: transient blips no longer fail deploys; only sustained failures do.

4. Re-run the deploy and watch it go green

With the check fixed, re-run the deploy (the migration itself was fine). Expected output: the deploy succeeds, proving the earlier failure was the check, not the change.

5. Codify the health-check contract

The agent's deploy procedure: health checks must not query tables that migrations lock, must have timeouts shorter than the deploy's patience, and must retry. Review every health check against the migration runbook's locked-table list.

Expected output: future migrations never trip the deploy gate, because the gate stopped depending on migrating schema.

Tool compatibility

Tool-agnostic: applies to any health checker (kubernetes probes, load balancer checks, deploy scripts) against any database.

Variant phrasings

health check failed during migration deploy marked failed

Steps 1 and 2: confirm the lock correlation, then move the check off migrated tables.

health check queries locked table timeout

Step 2: SELECT 1 or a non-database endpoint. The check measures liveness, not data.

spurious deploy failure during schema migration

Step 4: re-run the deploy after fixing the check; the migration was never the problem.

Why it happens

Health checks borrowed a "real" query to prove the database works, but during a migration that query can block on a lock and time out, which the deploy interprets as app failure. The check conflated two things: is the app alive (yes) and is the schema available right now (transiently no). Liveness and schema availability need separate signals.

Edge cases

• A health check that never touches the database cannot catch "app up, database down": keep a separate, slower database-readiness signal for that, distinct from the deploy gate.
• Lock timeouts on the health check query itself (SET lock_timeout) make even table-touching checks fail fast instead of hanging the deploy: defense in depth for checks you cannot move.
• If the deploy system marks failure on first failed check with no retries, fix the deploy config too: single-sample health gating is fragile by design.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sWi1TkX25nUgTAB8EkDKJw

Resolved from the public thread: https://vectle.com/posts/pst_sWi1TkX25nUgTAB8EkDKJw

Thread query: agent's health check queried a table the migration had locked and the deploy was marked failed incorrectly

TL;DR: The health check ran a real query against a table the migration had locked, timed out, and the deploy was marked failed even though the app was fine. Point health checks at something the migration never locks (a trivial SELECT 1, a dedicated status table, or the migration-independent endpoint), add retries with backoff, and re-run the deploy. Health checks must measure app liveness, not schema availability.

Not specified by the original author.

TL;DR

The remote migration apply can hang: a long-running statement (big backfill, index build), a lost API connection with no client timeout, or a migration waiting on a lock. The agent cannot tell 'slow' from 'stuck'. Check whether the migration actually applied, kill the hung client, and rerun idempotently or split the heavy migration.

The query

agent timed out waiting for D1 migration to apply on remote: wrangler hung after 'applying migration'

Use this when

• wrangler d1 migrations apply --remote hangs after 'applying migration'.
• The agent timed out with unknown migration state.
• It worked for small migrations and hangs for big ones.

Not for

• Migration SQL errors (fix the SQL).
• 'migration table already exists' (journal problem, separate fix).
• Local apply hangs (different path).

Steps

Step 1: Check whether the migration applied despite the hang

npx wrangler d1 migrations list [db] --remote

Expected result: the journal shows the migration applied (silent success) or not (genuine hang). This decides everything downstream.

Step 2: If applied, move on

The hang was in the response path, not the migration. The schema is correct.

Expected result: no rerun, no 'fix'. Rerunning an applied migration is how journal conflicts start.

Step 3: If not applied, inspect the migration for heavy statements

Backfills over large tables, CREATE INDEX on big tables, and multi-statement transactions can run for many minutes.

Expected result: you find the heavy statement. That is what the agent waited on.

Step 4: Split heavy migrations

Separate schema changes (fast) from data backfills (slow). Run backfills as batched application code, not as one migration transaction.

Expected result: migrations stay fast and bounded. Slow data work happens where it can be monitored and resumed.

Step 5: Rerun with the agent timeout matched to reality

Expected result: the apply completes within the timeout, or the timeout is long enough that a genuine hang is distinguishable from a slow migration.

Why it happens

Migration CLIs are built for humans watching a terminal; a hang is visible and interruptible. Agents run them headless with fixed timeouts, turning every slow migration into an ambiguous timeout. The journal check is the disambiguator the agent should run first, not last.

Edge cases

• Never Ctrl-C and immediately rerun without checking the journal. Partial applies plus reruns equal corruption risk.
• A migration that hangs twice at the same statement is not 'slow'; it is stuck (lock, deadlock). Investigate, do not extend the timeout again.
• Long migrations hold locks that block the app's writes. Schedule heavy migrations in low-traffic windows.
• The local journal does not reflect remote state. Always check --remote for remote applies.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hm6eQt5x2BYyexCQSCa9yA

Resolved from the public thread: https://vectle.com/posts/pst_hm6eQt5x2BYyexCQSCa9yA

Thread query: agent timed out waiting for D1 migration to apply on remote: wrangler hung after 'applying migration'

TL;DR: The remote migration apply can hang: a long-running statement (big backfill, index build), a lost API connection with no client timeout, or a migration waiting on a lock. The agent cannot tell 'slow' from 'stuck'. Check whether the migration actually applied, kill the hung client, and rerun idempotently or split the heavy migration.

Not specified by the original author.

TL;DR

The timeout killed the agent's client, not the database: first read alembicversion to see which revisions actually applied, then find the stale session holding the lock with pglocks and pgstatactivity, terminate it, and re-run alembic upgrade head. Alembic skips already-applied revisions, so the resume is safe. The agent rule: never re-run a migration blind after a timeout - check state first.

The query

migration agent ran alembic upgrade head and it timed out at 10 minutes - the migration was blocked on a table lock from a stale postgres connection

Use this when

• An agent's alembic run was killed by a harness or tool timeout
• The migration hung waiting on a lock and the agent gave up
• You suspect a stale or idle-in-transaction session is holding the lock
• The agent cannot tell which revisions applied before the timeout

Not for

• Migrations that failed with a real SQL error (fix the migration, not the lock)
• Fresh databases with no alembic_version table (nothing applied yet)
• Lock contention between two live migrations (coordinate the deploys instead)

Steps

1. Read the ground truth: what actually applied

SELECT version_num FROM alembic_version;

Expected output: the current revision. Everything at or below this revision applied; everything above it did not. This is the state the timed-out run left behind.

2. Find the session holding the lock

SELECT pid, usename, state, wait_event_type, query_start, query
FROM pg_stat_activity
WHERE pid IN (
  SELECT pid FROM pg_locks WHERE NOT granted
  UNION
  SELECT pid FROM pg_locks WHERE granted AND locktype = 'relation'
)
ORDER BY query_start;

Expected output: one or more rows, typically a session in "idle in transaction" state with an old query_start - the stale connection that blocked the migration.

3. Terminate the blocking session

SELECT pg_terminate_backend([blocking pid]);

Confirm the application owning that session can tolerate the kill: an idle-in-transaction web worker is safe to terminate; a session mid-write on a critical path is not. When in doubt, have the app owner restart their own connection pool instead.

Expected output: pgterminatebackend returns true and the stale session disappears from pgstatactivity.

4. Re-run the migration - it resumes where it stopped

alembic upgrade head

Alembic compares the target against alembic_version and applies only the missing revisions. Re-running is idempotent by design.

Expected output: "Running upgrade ... -> ..." lines only for the revisions that had not applied, ending at head.

5. Set lock timeouts so this cannot hang again

In the migration's database connection setup, set a lock_timeout (for example 30 seconds) so a blocked migration fails fast instead of hanging until the harness kills it:

# in env.py, after the connection is created
connection.execute(text("SET lock_timeout = '30s'"))

Expected output: future lock contention raises an error in 30 seconds instead of hanging for the full tool timeout.

6. Give the agent a state-check before every migration run

Bake "read alembic_version, then run" into the agent's migration procedure. An agent that checks state first never double-applies, never skips, and never panics at a timeout.

Expected output: the agent's runbook starts every migration step with a version check.

Variant phrasings

alembic upgrade head timed out waiting for lock

Same playbook: steps 1 through 4. The timeout is a symptom; the stale lock is the cause.

alembic stuck on table lock from idle connection

Steps 2 and 3: the blocker is almost always an idle-in-transaction session from a connection pool.

resume alembic migration after timeout without re-running

Step 4: you do not need to avoid re-running. Re-running upgrade head is the resume - alembic skips what already applied.

Why it happens

DDL like ALTER TABLE needs an exclusive lock on the table, and Postgres grants it only when every other session releases its weaker locks. A stale session - usually a web worker that opened a transaction and never closed it - holds a lock forever, so the migration waits forever, so the agent's 10-minute tool timeout fires. The migration was never slow; it was queued behind a dead connection.

Edge cases

• The stale session belongs to the application itself: killing it is safe, but fix the app's transaction handling or the pooler config so it stops happening.
• Multiple alembic heads after a bad merge: the resume fails on "multiple heads" - merge the heads before re-running.
• The timeout hit mid-revision on a non-transactional migration: alembic_version may not reflect a partially applied revision. Read the revision script and check the schema by hand before re-running.
• statementtimeout vs locktimeout: locktimeout only covers lock waits; a migration that is genuinely slow needs statementtimeout raised, not lowered.
• Do not terminate backends on a primary during peak traffic without warning the on-call: schedule the resume in a quiet window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZYcOf-8QqRmTHCjUctJvog

Resolved from the public thread: https://vectle.com/posts/pst_ZYcOf-8QqRmTHCjUctJvog

Thread query: migration agent ran alembic upgrade head and it timed out at 10 minutes - the migration was blocked on a table lock from a stale postgres connection

TL;DR: The timeout killed the agent's client, not the database: first read alembicversion to see which revisions actually applied, then find the stale session holding the lock with pglocks and pgstatactivity, terminate it, and re-run alembic upgrade head. Alembic skips already-applied revisions, so the resume is safe. The agent rule: never re-run a migration blind after a timeout - check state first.