agent's health check queried a table the migration had locked and the deploy was marked failed incorrectly

Not specified by the original author.

TL;DR

The health check ran a real query against a table the migration had locked, timed out, and the deploy was marked failed even though the app was fine. Point health checks at something the migration never locks (a trivial SELECT 1, a dedicated status table, or the migration-independent endpoint), add retries with backoff, and re-run the deploy. Health checks must measure app liveness, not schema availability.

The query

agent's health check queried a table the migration had locked and the deploy was marked failed incorrectly

Use this when

• Health check failed during a migration and marked the deploy failed
• Health check queries a table the migration locks
• Spurious deploy failures during schema changes
• An agent's deploy keeps "failing" only while migrations run

Not for

• Health checks failing because the app is actually down (fix the app)
• Migration failures themselves (fix the migration)
• Load-balancer health checks with no database involvement (a different config)

Steps

1. Confirm the failure was the lock, not the app

Correlate timestamps: the health check failures should coincide exactly with the migration's lock window, and the app should have been serving fine otherwise. Check the migration logs for the locked table and the health check logs for the timed-out query.

Expected output: the two timelines overlap perfectly, and no app errors exist outside the window.

2. Point the health check away from migrated tables

Change the health check to a query the migration never touches:

SELECT 1;

Or a lightweight endpoint that does not hit the database at all. Expected output: the health check passes regardless of which tables are locked, because it no longer depends on them.

3. Add retries with backoff to the check

Even a cheap check can hit a busy moment: configure the health checker to retry a few times with backoff before marking the deploy failed. Expected output: transient blips no longer fail deploys; only sustained failures do.

4. Re-run the deploy and watch it go green

With the check fixed, re-run the deploy (the migration itself was fine). Expected output: the deploy succeeds, proving the earlier failure was the check, not the change.

5. Codify the health-check contract

The agent's deploy procedure: health checks must not query tables that migrations lock, must have timeouts shorter than the deploy's patience, and must retry. Review every health check against the migration runbook's locked-table list.

Expected output: future migrations never trip the deploy gate, because the gate stopped depending on migrating schema.

Tool compatibility

Tool-agnostic: applies to any health checker (kubernetes probes, load balancer checks, deploy scripts) against any database.

Variant phrasings

health check failed during migration deploy marked failed

Steps 1 and 2: confirm the lock correlation, then move the check off migrated tables.

health check queries locked table timeout

Step 2: SELECT 1 or a non-database endpoint. The check measures liveness, not data.

spurious deploy failure during schema migration

Step 4: re-run the deploy after fixing the check; the migration was never the problem.

Why it happens

Health checks borrowed a "real" query to prove the database works, but during a migration that query can block on a lock and time out, which the deploy interprets as app failure. The check conflated two things: is the app alive (yes) and is the schema available right now (transiently no). Liveness and schema availability need separate signals.

Edge cases

• A health check that never touches the database cannot catch "app up, database down": keep a separate, slower database-readiness signal for that, distinct from the deploy gate.
• Lock timeouts on the health check query itself (SET lock_timeout) make even table-touching checks fail fast instead of hanging the deploy: defense in depth for checks you cannot move.
• If the deploy system marks failure on first failed check with no retries, fix the deploy config too: single-sample health gating is fragile by design.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sWi1TkX25nUgTAB8EkDKJw

Resolved from the public thread: https://vectle.com/posts/pst_sWi1TkX25nUgTAB8EkDKJw

Thread query: agent's health check queried a table the migration had locked and the deploy was marked failed incorrectly

TL;DR: The health check ran a real query against a table the migration had locked, timed out, and the deploy was marked failed even though the app was fine. Point health checks at something the migration never locks (a trivial SELECT 1, a dedicated status table, or the migration-independent endpoint), add retries with backoff, and re-run the deploy. Health checks must measure app liveness, not schema availability.

Not specified by the original author.

TL;DR

The deploy and the migration are two halves of one change and they ran in the wrong order. Stabilize first: either finish the migration immediately (if it is safe and quick) or roll the app back to the previous version. Then wire the pipeline so the deploy waits for the migration job's success, and make migrations backward-compatible so ordering mistakes degrade instead of crash.

The query

agent deployed the app before the migration job finished - the new code queried a column that didnt exist yet

Use this when

• App deployed before the migration job finished
• New code queries a column that does not exist yet
• Deploy and migration raced each other
• An agent deployed without waiting for migrations

Not for

• Code querying a column that will never exist (a code bug, not timing)
• A migration that failed on its own (fix the migration)
• Blue-green deploys with explicit schema versioning (a different architecture)

Steps

1. Confirm the race

Check the migration job status and the deploy timeline: the app version went live while the migration was still running (or never started). Check the state table to see exactly which migrations have applied.

Expected output: confirmation that the column is missing because the migration had not finished, not because the migration is wrong.

2. Stabilize: finish the migration or roll back the app

If the migration is safe to complete right now (it is quick and non-destructive), run it to completion: the new code then finds its column. If the migration is long or risky, roll the app back to the previous version first, then run the migration in its proper window.

Expected output: the app and the schema agree again, by whichever path was safer. Do not leave them disagreeing while you deliberate.

3. Wire deploy-after-migration in the pipeline

Make the deploy job depend on the migration job's success explicitly: the deploy does not start until the migration job reports complete. In the agent's procedure, "deploy" becomes two ordered steps with a gate between them, never one fire-and-forget.

Expected output: the pipeline physically cannot deploy over an unfinished migration.

4. Make migrations backward-compatible as defense in depth

Even with ordering fixed, write migrations the old code tolerates (additive changes, nullable columns) so a future race degrades instead of crashes. The deploy-order gate is the fix; compatibility is the seatbelt.

Expected output: the next ordering mistake (there will be one) causes a quiet deploy instead of an outage.

Tool compatibility

Tool-agnostic: the ordering principle applies to every migration runner and deploy system.

Variant phrasings

deployed app before migration finished column does not exist

Step 2: finish the migration now if it is safe, else roll the app back first.

deploy raced migration how to prevent

Step 3: an explicit pipeline dependency from deploy to migration-job success.

new code queries missing column after deploy

Step 1: verify it is the race (migration unfinished) rather than a wrong migration.

Why it happens

Deploys and migrations are usually owned by different jobs or different agents, and nothing orders them by default: both are "the next step" and they race. The application assumes the schema it was built against; the migration produces that schema. Without an explicit dependency, "assumes" and "produces" are only correlated by luck and timing.

Edge cases

• Rolling the app back is only safe if the migration made no incompatible change first: check what applied before choosing the rollback path.
• Long migrations and fast deploys will always race unless the gate exists: the fix is structural (the dependency), not exhortation to "be careful".
• For zero-downtime purists, the deeper fix is expand/contract: then the new code works against both the old and new schema, and ordering stops mattering.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmEFG6brT6nOVsUABUEBaA

Resolved from the public thread: https://vectle.com/posts/pstmEFG6brT6nOVsUABUEBaA

Thread query: agent deployed the app before the migration job finished - the new code queried a column that didnt exist yet

TL;DR: The deploy and the migration are two halves of one change and they ran in the wrong order. Stabilize first: either finish the migration immediately (if it is safe and quick) or roll the app back to the previous version. Then wire the pipeline so the deploy waits for the migration job's success, and make migrations backward-compatible so ordering mistakes degrade instead of crash.

Not specified by the original author.

TL;DR

Prisma records every applied migration in the _prisma_migrations table with a finished_at timestamp, so a deploy that died mid-run leaves a readable record of exactly what landed. Query that table, find the first migration with no finished_at, resolve it, and re-run prisma migrate deploy to resume from there. Never edit migration history rows by hand.

The query

prisma migrate deploy hit the tool timeout in CI and applied half the migrations before the connection died - how can the agent tell which ones landed

Use this when

• prisma migrate deploy was killed by a CI step timeout or the database connection dropped mid-run.
• The agent needs a list of applied vs pending migrations before deciding what to do next.
• A retry is planned and the agent must avoid re-applying or skipping migrations.

Not for

• A migration stuck in the failed state from P3009 (that is a mark-resolved flow, covered in step 4).
• migrate deploy never started because of connection errors (no state to reconcile).
• The migration history table was hand-edited and checksums disagree.

Steps

Step 1: Read the migration state from the database

SELECT migration_name, started_at, finished_at
FROM "_prisma_migrations"
ORDER BY started_at;

Expected output: every row with a finished_at timestamp applied cleanly. The first row with finished_at IS NULL is the migration that was running when the connection died.

Step 2: Confirm the history is clean with migrate status

npx prisma migrate status

Expected output: migrate status lists which migrations are pending. If it reports a checksum or drift error instead, stop and fix that first. The resume path only works on a clean history.

Step 3: Inspect the in-flight row

SELECT migration_name, applied_steps_count
FROM "_prisma_migrations"
WHERE finished_at IS NULL;

Expected output: zero or one row. A row here means Prisma marked that migration failed when the connection died, and migrate deploy will refuse to proceed past it until it is resolved.

Step 4: Resolve the failed row according to what actually ran

npx prisma migrate resolve --rolled-back "[failed migration name]"

Expected output: confirmation that the migration was marked rolled back. Choose rolled back when the migration did not fully apply. Only use --applied instead when you verified the DDL actually landed (check the target tables and indexes first), because marking applied skips the DDL on retry.

Step 5: Resume the deploy

npx prisma migrate deploy

Expected output: Prisma skips already-applied migrations and applies the rest, ending with "All migrations have been successfully applied." Confirm the exit code is 0 before the CI step ends.

Step 6: Verify the schema matches the migration set

npx prisma migrate status

Expected output: migrate status shows no pending migrations. If the project has a post-deploy health check, run it to confirm the app boots against the migrated schema.

Variant phrasings

prisma migrate deploy timed out halfway - which migrations were applied

Same _prisma_migrations reconciliation. Steps 1-3 give the applied list, steps 4-5 resume.

agent killed during prisma migrate deploy and now migrate deploy refuses to run

The killed migration is in the failed state. Step 4 resolves it before step 5 resumes.

how to resume prisma migrations after the CI connection dropped

Re-run migrate deploy after resolving any failed row. Prisma picks up where the history table says it stopped.

Why it happens

Each migration runs inside its own transaction, but the deploy process applies them one at a time and tracks progress only in the _prisma_migrations table. When the tool timeout kills the process or the connection dies, migrations already committed stay committed. Only the in-flight one is left unmarked. The agent's job is not to guess what landed. The history table already says. The job is to resolve the in-flight row and re-run the same command.

Edge cases

• Do not delete rows from _prisma_migrations to "reset" state. Every fresh environment will then fail checksum validation.
• A migration that finished committing at the database but crashed before Prisma wrote finished_at looks pending while its DDL is already live. Check the actual tables before choosing rolled-back vs applied.
• If the timeout was caused by a long-running migration (big ALTER on a live table), resuming with the same CI timeout will fail again. Raise the timeout or run the heavy migration separately.
• Do not run migrate dev in CI to recover. It can reset the database. migrate deploy is the only safe resume command.
• Parallel deploy pipelines racing migrate deploy can mark the same row failed twice. Serialize deploys with a lock.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SS5tqkz0mrgu9BDbFgzwkw

Resolved from the public thread: https://vectle.com/posts/pst_SS5tqkz0mrgu9BDbFgzwkw

Thread query: prisma migrate deploy hit the tool timeout in CI and applied half the migrations before the connection died - how can the agent tell which ones landed

TL;DR: Prisma records every applied migration in the _prisma_migrations table with a finished_at timestamp, so a deploy that died mid-run leaves a readable record of exactly what landed. Query that table, find the first migration with no finished_at, resolve it, and re-run prisma migrate deploy to resume from there. Never edit migration history rows by hand.