agent deployed the app before the migration job finished - the new code queried a column that didnt exist yet

TL;DR

The deploy and the migration are two halves of one change and they ran in the wrong order. Stabilize first: either finish the migration immediately (if it is safe and quick) or roll the app back to the previous version. Then wire the pipeline so the deploy waits for the migration job's success, and make migrations backward-compatible so ordering mistakes degrade instead of crash.

The query

agent deployed the app before the migration job finished - the new code queried a column that didnt exist yet

Use this when

• App deployed before the migration job finished
• New code queries a column that does not exist yet
• Deploy and migration raced each other
• An agent deployed without waiting for migrations

Not for

• Code querying a column that will never exist (a code bug, not timing)
• A migration that failed on its own (fix the migration)
• Blue-green deploys with explicit schema versioning (a different architecture)

Steps

1. Confirm the race

Check the migration job status and the deploy timeline: the app version went live while the migration was still running (or never started). Check the state table to see exactly which migrations have applied.

Expected output: confirmation that the column is missing because the migration had not finished, not because the migration is wrong.

2. Stabilize: finish the migration or roll back the app

If the migration is safe to complete right now (it is quick and non-destructive), run it to completion: the new code then finds its column. If the migration is long or risky, roll the app back to the previous version first, then run the migration in its proper window.

Expected output: the app and the schema agree again, by whichever path was safer. Do not leave them disagreeing while you deliberate.

3. Wire deploy-after-migration in the pipeline

Make the deploy job depend on the migration job's success explicitly: the deploy does not start until the migration job reports complete. In the agent's procedure, "deploy" becomes two ordered steps with a gate between them, never one fire-and-forget.

Expected output: the pipeline physically cannot deploy over an unfinished migration.

4. Make migrations backward-compatible as defense in depth

Even with ordering fixed, write migrations the old code tolerates (additive changes, nullable columns) so a future race degrades instead of crashes. The deploy-order gate is the fix; compatibility is the seatbelt.

Expected output: the next ordering mistake (there will be one) causes a quiet deploy instead of an outage.

Tool compatibility

Tool-agnostic: the ordering principle applies to every migration runner and deploy system.

Variant phrasings

deployed app before migration finished column does not exist

Step 2: finish the migration now if it is safe, else roll the app back first.

deploy raced migration how to prevent

Step 3: an explicit pipeline dependency from deploy to migration-job success.

new code queries missing column after deploy

Step 1: verify it is the race (migration unfinished) rather than a wrong migration.

Why it happens

Deploys and migrations are usually owned by different jobs or different agents, and nothing orders them by default: both are "the next step" and they race. The application assumes the schema it was built against; the migration produces that schema. Without an explicit dependency, "assumes" and "produces" are only correlated by luck and timing.

Edge cases

• Rolling the app back is only safe if the migration made no incompatible change first: check what applied before choosing the rollback path.
• Long migrations and fast deploys will always race unless the gate exists: the fix is structural (the dependency), not exhortation to "be careful".
• For zero-downtime purists, the deeper fix is expand/contract: then the new code works against both the old and new schema, and ordering stops mattering.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmEFG6brT6nOVsUABUEBaA