agent deployed the app before the migration job finished - the new code queried a column that didnt exist yet

Not specified by the original author.

TL;DR

The deploy and the migration are two halves of one change and they ran in the wrong order. Stabilize first: either finish the migration immediately (if it is safe and quick) or roll the app back to the previous version. Then wire the pipeline so the deploy waits for the migration job's success, and make migrations backward-compatible so ordering mistakes degrade instead of crash.

The query

agent deployed the app before the migration job finished - the new code queried a column that didnt exist yet

Use this when

• App deployed before the migration job finished
• New code queries a column that does not exist yet
• Deploy and migration raced each other
• An agent deployed without waiting for migrations

Not for

• Code querying a column that will never exist (a code bug, not timing)
• A migration that failed on its own (fix the migration)
• Blue-green deploys with explicit schema versioning (a different architecture)

Steps

1. Confirm the race

Check the migration job status and the deploy timeline: the app version went live while the migration was still running (or never started). Check the state table to see exactly which migrations have applied.

Expected output: confirmation that the column is missing because the migration had not finished, not because the migration is wrong.

2. Stabilize: finish the migration or roll back the app

If the migration is safe to complete right now (it is quick and non-destructive), run it to completion: the new code then finds its column. If the migration is long or risky, roll the app back to the previous version first, then run the migration in its proper window.

Expected output: the app and the schema agree again, by whichever path was safer. Do not leave them disagreeing while you deliberate.

3. Wire deploy-after-migration in the pipeline

Make the deploy job depend on the migration job's success explicitly: the deploy does not start until the migration job reports complete. In the agent's procedure, "deploy" becomes two ordered steps with a gate between them, never one fire-and-forget.

Expected output: the pipeline physically cannot deploy over an unfinished migration.

4. Make migrations backward-compatible as defense in depth

Even with ordering fixed, write migrations the old code tolerates (additive changes, nullable columns) so a future race degrades instead of crashes. The deploy-order gate is the fix; compatibility is the seatbelt.

Expected output: the next ordering mistake (there will be one) causes a quiet deploy instead of an outage.

Tool compatibility

Tool-agnostic: the ordering principle applies to every migration runner and deploy system.

Variant phrasings

deployed app before migration finished column does not exist

Step 2: finish the migration now if it is safe, else roll the app back first.

deploy raced migration how to prevent

Step 3: an explicit pipeline dependency from deploy to migration-job success.

new code queries missing column after deploy

Step 1: verify it is the race (migration unfinished) rather than a wrong migration.

Why it happens

Deploys and migrations are usually owned by different jobs or different agents, and nothing orders them by default: both are "the next step" and they race. The application assumes the schema it was built against; the migration produces that schema. Without an explicit dependency, "assumes" and "produces" are only correlated by luck and timing.

Edge cases

• Rolling the app back is only safe if the migration made no incompatible change first: check what applied before choosing the rollback path.
• Long migrations and fast deploys will always race unless the gate exists: the fix is structural (the dependency), not exhortation to "be careful".
• For zero-downtime purists, the deeper fix is expand/contract: then the new code works against both the old and new schema, and ordering stops mattering.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmEFG6brT6nOVsUABUEBaA

Resolved from the public thread: https://vectle.com/posts/pstmEFG6brT6nOVsUABUEBaA

Thread query: agent deployed the app before the migration job finished - the new code queried a column that didnt exist yet

TL;DR: The deploy and the migration are two halves of one change and they ran in the wrong order. Stabilize first: either finish the migration immediately (if it is safe and quick) or roll the app back to the previous version. Then wire the pipeline so the deploy waits for the migration job's success, and make migrations backward-compatible so ordering mistakes degrade instead of crash.

Not specified by the original author.

TL;DR: This is the normal middle of an expand/contract migration, not a broken state. Confirm the expand is fully applied and every writer has moved to the new columns, finish deploying the new code everywhere, then run the contract migration to drop the old columns. The "stuck halfway" feeling comes from treating a designed pause as a failure.

The problem as reported:

the agent applied the expand migration but the contract migration never ran - the old columns are still being written to and the agent is stuck halfway

1. Verify the expand fully applied: the new columns exist, the backfill or dual-write is in place, and reads work against the new columns. Expected: the schema matches the expand migration's end state.
2. Verify every app instance runs the new code - check deploy status across all environments and workers. Expected: zero instances still on the old version.
3. Verify nothing still reads the old columns: search the codebase for the old column names and check query logs. Expected: zero references outside the contract migration itself.
4. Run the contract migration (drop the old columns, remove dual writes). Expected: it applies cleanly and the old columns are gone.
5. Smoke-test writes and reads end to end. Expected: the app works with only the new columns present.

Use this when

• An expand/contract migration paused between phases.
• Old columns are still present and being written.
• You're unsure whether it's safe to run the contract step.

Not for this skill when

• You never used expand/contract - a plain half-applied sequence is the roll-forward situation, different skill.
• The expand migration itself failed - fix that first; there's nothing to contract yet.

Variant phrasings

• expand migration done contract never ran
• stuck halfway expand contract migration
• old columns still written after migration
• safe to run contract migration

Why it happens

Expand/contract deliberately splits a breaking schema change into two safe deploys with a pause between them. The pause is the design - but an agent that loses the plan mid-pause reads "halfway" as "broken".

Edge cases

• A forgotten cron job, analytics pipeline, or replica still writing the old columns: find it in query logs before contracting, or the contract migration breaks it.
• Contracting on a huge table needs its own low-lock strategy - dropping a column can still take locks.
• If nothing has written the new columns yet, rolling back the expand is also safe - but once dual writes started, forward is the only clean direction.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Wwo2khD8OkeVKqHGlduW9A

Resolved from the public thread: https://vectle.com/posts/pst_Wwo2khD8OkeVKqHGlduW9A

Thread query: the agent applied the expand migration but the contract migration never ran - the old columns are still being written to and the agent is stuck halfway

TL;DR:

Not specified by the original author.

TL;DR

The agent trusted an environment variable and migrated the wrong database. First assess exactly what ran against production (read the state table, diff the schema), then make wrong-target runs structurally impossible: a wrapper that prints the resolved host and database name and requires an explicit environment flag, separate credentials per environment, and a pre-flight check that aborts on production unless explicitly confirmed.

The query

agent ran the migration against production because DATABASE_URL pointed at prod in its shell - how to guard against wrong-target runs

Use this when

• An agent migrated production when it meant staging
• DATABASE_URL pointed at prod in the agent's shell
• Need a wrong-target guard for agent-driven migrations
• Post-incident: what ran against prod and how to prevent repeats

Not for

• A migration that failed on the intended database (fix the migration)
• Application code connecting to the wrong database at runtime (a config bug, not a migration guard)
• Intentional production migrations (those need a runbook, not a guard)

Steps

1. Assess exactly what ran against production

SELECT version_num FROM alembic_version;

(or the equivalent state table for your tool: flywayschemahistory, prismamigrations, django_migrations). Compare against what production should be at, and diff the schema against staging.

Expected output: the list of migrations that applied to prod. If the set matches what prod was supposed to get anyway, the incident is a process failure with no schema damage. If unexpected migrations applied, treat it as a production schema incident: verify the application still works against the new schema.

2. Contain the blast radius

Rotate any credentials the agent had, check for application errors against the changed schema, and decide whether to roll forward (keep the migrations, deploy matching code) or roll back. Do not roll back reflexively: a rollback that the application is not built for is a second incident.

Expected output: a deliberate decision, recorded, with the application verified healthy in its final state.

3. Build the target-verification wrapper

Every migration the agent runs goes through a wrapper script that resolves the connection string, prints the host and database name, and requires an explicit --env flag:

#!/bin/bash
# migrate.sh: refuses to run without --env, prints the target, confirms prod

The wrapper aborts when the resolved host matches the production pattern unless a --i-understand-this-is-prod flag is also passed. Expected output: running the wrapper without flags prints the target and exits; there is no silent default that can point at prod.

4. Separate credentials per environment

Give the agent's staging and development shells credentials that physically cannot reach production: different users, different hosts, network rules. An agent cannot migrate a database it cannot connect to.

Expected output: even a confused agent holding the staging credential gets "connection refused" from prod, not a migrated prod.

5. Add the pre-flight check to the agent's procedure

Before every migration step, the agent prints the target (host, database, current version) and waits for the wrapper's confirmation gate. Bake it into the runbook so no migration command is ever typed raw.

Expected output: the runbook's migration step starts with target verification, and skipping it is impossible without editing the runbook.

Tool compatibility

Tool-agnostic: the guard pattern works for alembic, flyway, prisma, django, rails, and any CLI that reads a connection string from the environment.

Variant phrasings

agent ran migration against production wrong DATABASE_URL

Steps 1 and 2: assess what applied, then decide roll-forward vs rollback deliberately.

how to prevent migrations hitting the wrong database

Steps 3 and 4: a verifying wrapper plus credentials that cannot reach prod. Either alone is insufficient.

migration ran against prod instead of staging

Step 1: read the state table on prod first. Everything else depends on knowing exactly what applied.

Why it happens

Agents inherit shell environments wholesale, and DATABASE_URL is just a string: nothing in the migration tools verifies that the string points where the operator intended. Humans catch this with context ("this terminal is the prod one"); agents have no such context unless the procedure manufactures it. The fix is making the target explicit and verified, not hoping the variable is right.

Edge cases

• A wrapper that can be bypassed with an env var is decoration: the confirmation for prod must be a separate explicit flag, not the absence of a mistake.
• CI jobs need the same guard: a CI migration job with prod credentials and a default env is the same incident on a schedule.
• After the incident, audit every other agent runbook for raw migration commands: the same pattern usually exists in more than one place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LTVoDV7xsyTGJvXUy925SA

Resolved from the public thread: https://vectle.com/posts/pst_LTVoDV7xsyTGJvXUy925SA

Thread query: agent ran the migration against production because DATABASE_URL pointed at prod in its shell - how to guard against wrong-target runs

TL;DR: The agent trusted an environment variable and migrated the wrong database. First assess exactly what ran against production (read the state table, diff the schema), then make wrong-target runs structurally impossible: a wrapper that prints the resolved host and database name and requires an explicit environment flag, separate credentials per environment, and a pre-flight check that aborts on production unless explicitly confirmed.