agent's health check queried a table the migration had locked and the deploy was marked failed incorrectly

TL;DR

The health check ran a real query against a table the migration had locked, timed out, and the deploy was marked failed even though the app was fine. Point health checks at something the migration never locks (a trivial SELECT 1, a dedicated status table, or the migration-independent endpoint), add retries with backoff, and re-run the deploy. Health checks must measure app liveness, not schema availability.

The query

agent's health check queried a table the migration had locked and the deploy was marked failed incorrectly

Use this when

• Health check failed during a migration and marked the deploy failed
• Health check queries a table the migration locks
• Spurious deploy failures during schema changes
• An agent's deploy keeps "failing" only while migrations run

Not for

• Health checks failing because the app is actually down (fix the app)
• Migration failures themselves (fix the migration)
• Load-balancer health checks with no database involvement (a different config)

Steps

1. Confirm the failure was the lock, not the app

Correlate timestamps: the health check failures should coincide exactly with the migration's lock window, and the app should have been serving fine otherwise. Check the migration logs for the locked table and the health check logs for the timed-out query.

Expected output: the two timelines overlap perfectly, and no app errors exist outside the window.

2. Point the health check away from migrated tables

Change the health check to a query the migration never touches:

SELECT 1;

Or a lightweight endpoint that does not hit the database at all. Expected output: the health check passes regardless of which tables are locked, because it no longer depends on them.

3. Add retries with backoff to the check

Even a cheap check can hit a busy moment: configure the health checker to retry a few times with backoff before marking the deploy failed. Expected output: transient blips no longer fail deploys; only sustained failures do.

4. Re-run the deploy and watch it go green

With the check fixed, re-run the deploy (the migration itself was fine). Expected output: the deploy succeeds, proving the earlier failure was the check, not the change.

5. Codify the health-check contract

The agent's deploy procedure: health checks must not query tables that migrations lock, must have timeouts shorter than the deploy's patience, and must retry. Review every health check against the migration runbook's locked-table list.

Expected output: future migrations never trip the deploy gate, because the gate stopped depending on migrating schema.

Tool compatibility

Tool-agnostic: applies to any health checker (kubernetes probes, load balancer checks, deploy scripts) against any database.

Variant phrasings

health check failed during migration deploy marked failed

Steps 1 and 2: confirm the lock correlation, then move the check off migrated tables.

health check queries locked table timeout

Step 2: SELECT 1 or a non-database endpoint. The check measures liveness, not data.

spurious deploy failure during schema migration

Step 4: re-run the deploy after fixing the check; the migration was never the problem.

Why it happens

Health checks borrowed a "real" query to prove the database works, but during a migration that query can block on a lock and time out, which the deploy interprets as app failure. The check conflated two things: is the app alive (yes) and is the schema available right now (transiently no). Liveness and schema availability need separate signals.

Edge cases

• A health check that never touches the database cannot catch "app up, database down": keep a separate, slower database-readiness signal for that, distinct from the deploy gate.
• Lock timeouts on the health check query itself (SET lock_timeout) make even table-touching checks fail fast instead of hanging the deploy: defense in depth for checks you cannot move.
• If the deploy system marks failure on first failed check with no retries, fix the deploy config too: single-sample health gating is fragile by design.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sWi1TkX25nUgTAB8EkDKJw