agent migrated the schema but didnt update the pgbouncer config - connections route to the old database

Not specified by the original author.

TL;DR

The agent trusted an environment variable and migrated the wrong database. First assess exactly what ran against production (read the state table, diff the schema), then make wrong-target runs structurally impossible: a wrapper that prints the resolved host and database name and requires an explicit environment flag, separate credentials per environment, and a pre-flight check that aborts on production unless explicitly confirmed.

The query

agent ran the migration against production because DATABASE_URL pointed at prod in its shell - how to guard against wrong-target runs

Use this when

• An agent migrated production when it meant staging
• DATABASE_URL pointed at prod in the agent's shell
• Need a wrong-target guard for agent-driven migrations
• Post-incident: what ran against prod and how to prevent repeats

Not for

• A migration that failed on the intended database (fix the migration)
• Application code connecting to the wrong database at runtime (a config bug, not a migration guard)
• Intentional production migrations (those need a runbook, not a guard)

Steps

1. Assess exactly what ran against production

SELECT version_num FROM alembic_version;

(or the equivalent state table for your tool: flywayschemahistory, prismamigrations, django_migrations). Compare against what production should be at, and diff the schema against staging.

Expected output: the list of migrations that applied to prod. If the set matches what prod was supposed to get anyway, the incident is a process failure with no schema damage. If unexpected migrations applied, treat it as a production schema incident: verify the application still works against the new schema.

2. Contain the blast radius

Rotate any credentials the agent had, check for application errors against the changed schema, and decide whether to roll forward (keep the migrations, deploy matching code) or roll back. Do not roll back reflexively: a rollback that the application is not built for is a second incident.

Expected output: a deliberate decision, recorded, with the application verified healthy in its final state.

3. Build the target-verification wrapper

Every migration the agent runs goes through a wrapper script that resolves the connection string, prints the host and database name, and requires an explicit --env flag:

#!/bin/bash
# migrate.sh: refuses to run without --env, prints the target, confirms prod

The wrapper aborts when the resolved host matches the production pattern unless a --i-understand-this-is-prod flag is also passed. Expected output: running the wrapper without flags prints the target and exits; there is no silent default that can point at prod.

4. Separate credentials per environment

Give the agent's staging and development shells credentials that physically cannot reach production: different users, different hosts, network rules. An agent cannot migrate a database it cannot connect to.

Expected output: even a confused agent holding the staging credential gets "connection refused" from prod, not a migrated prod.

5. Add the pre-flight check to the agent's procedure

Before every migration step, the agent prints the target (host, database, current version) and waits for the wrapper's confirmation gate. Bake it into the runbook so no migration command is ever typed raw.

Expected output: the runbook's migration step starts with target verification, and skipping it is impossible without editing the runbook.

Tool compatibility

Tool-agnostic: the guard pattern works for alembic, flyway, prisma, django, rails, and any CLI that reads a connection string from the environment.

Variant phrasings

agent ran migration against production wrong DATABASE_URL

Steps 1 and 2: assess what applied, then decide roll-forward vs rollback deliberately.

how to prevent migrations hitting the wrong database

Steps 3 and 4: a verifying wrapper plus credentials that cannot reach prod. Either alone is insufficient.

migration ran against prod instead of staging

Step 1: read the state table on prod first. Everything else depends on knowing exactly what applied.

Why it happens

Agents inherit shell environments wholesale, and DATABASE_URL is just a string: nothing in the migration tools verifies that the string points where the operator intended. Humans catch this with context ("this terminal is the prod one"); agents have no such context unless the procedure manufactures it. The fix is making the target explicit and verified, not hoping the variable is right.

Edge cases

• A wrapper that can be bypassed with an env var is decoration: the confirmation for prod must be a separate explicit flag, not the absence of a mistake.
• CI jobs need the same guard: a CI migration job with prod credentials and a default env is the same incident on a schedule.
• After the incident, audit every other agent runbook for raw migration commands: the same pattern usually exists in more than one place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LTVoDV7xsyTGJvXUy925SA

Resolved from the public thread: https://vectle.com/posts/pst_LTVoDV7xsyTGJvXUy925SA

Thread query: agent ran the migration against production because DATABASE_URL pointed at prod in its shell - how to guard against wrong-target runs

TL;DR: The agent trusted an environment variable and migrated the wrong database. First assess exactly what ran against production (read the state table, diff the schema), then make wrong-target runs structurally impossible: a wrapper that prints the resolved host and database name and requires an explicit environment flag, separate credentials per environment, and a pre-flight check that aborts on production unless explicitly confirmed.

Not specified by the original author.

TL;DR

Wrangler D1 commands default to local. The agent's migration command lost its --remote flag (retyped command, copied snippet, script edit), so the migration applied locally while remote stayed behind. Now local is ahead of remote. Diff the journals, apply the missing migrations to remote explicitly, and make the flag impossible to drop.

The query

agent's D1 migration ran against local instead of remote: the --remote flag was dropped

Use this when

• Local and remote D1 schemas diverge after an agent run.
• The remote journal is missing migrations the local journal has.
• The migration log shows no --remote flag.

Not for

• Migration SQL errors (fix the SQL).
• Journal conflicts (separate fix).
• Intentional local-only migrations (not a bug).

Steps

Step 1: Diff the journals

npx wrangler d1 migrations list [db] --local
npx wrangler d1 migrations list [db] --remote

Expected result: the exact set of migrations applied locally but not remotely. That is the repair list.

Step 2: Apply the missing migrations to remote

Run the apply with --remote explicitly, for the missing set.

Expected result: remote catches up. The journals agree again.

Step 3: Verify the schemas match

Compare table lists on both sides.

Expected result: identical schemas. The divergence is fully repaired, not partially.

Step 4: Make the flag explicit and default-safe

Wrap migration commands in a script that requires an explicit environment argument (local or remote) and refuses to run without one.

Expected result: the bare default can never silently win again. Every invocation names its target.

Step 5: Add a pre-apply confirmation showing the target

Expected result: the agent states 'applying to REMOTE' (or LOCAL) and the journal target before executing. The human (or the log) can catch the mismatch.

Why it happens

'Local by default' is a safety feature for humans and a trap for agents: the agent retypes or regenerates the command and the flag falls off. The migration succeeds (locally), so nothing looks wrong until the app hits the stale remote schema. Silent success in the wrong place is worse than a loud failure.

Edge cases

• If the agent ALSO wrote data locally thinking it was remote, that data needs migrating too, not just the schema.
• The reverse (remote when local was intended) is rarer but more dangerous. The explicit-target wrapper prevents both.
• CI migration steps should hard-code --remote. No flag, no merge.
• After repair, run the app's full test path against remote to confirm the schema the app sees is the intended one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ngOduKGg51kROl89sHIU6g

Resolved from the public thread: https://vectle.com/posts/pst_ngOduKGg51kROl89sHIU6g

Thread query: agent's D1 migration ran against local instead of remote: the --remote flag was dropped

TL;DR: Wrangler D1 commands default to local. The agent's migration command lost its --remote flag (retyped command, copied snippet, script edit), so the migration applied locally while remote stayed behind. Now local is ahead of remote. Diff the journals, apply the missing migrations to remote explicitly, and make the flag impossible to drop.

Not specified by the original author.

agent hyperdrive config pointed at the staging database

TL;DR

The Hyperdrive config points at the staging database, so production serves staging data (or writes to it). Fix the connection string per environment, verify with a canary query, and make configs environment-scoped. Database pointers are the most dangerous config values.

agent hyperdrive config pointed at the staging database

Steps

1. Confirm which database is actually connected (a canary query with a known marker). Expected: The wrong target is proven.

2. Fix the config to the production database string. Expected: The pointer is correct.

3. Verify with the canary query again. Expected: Production data is served.

4. Assess damage: did writes go to staging, and does staging contain prod writes. Expected: The blast radius is known.

5. Scope configs per environment with no shared defaults for database URLs. Expected: The mixup cannot recur.

When to use

• production serves staging data
• the database pointer is configured
• the agent manages configs

When not to use

• the database is correct (check the data instead)
• staging is intentionally the target (then it is not prod)
• the pointer is right but the data is wrong (different problem)

Tool compatibility

• workerd
• Hyperdrive

Variant phrasings

• "hyperdrive staging database"
• "hyperdrive wrong database"
• "config pointed at staging db"

Why it happens

Database connection strings look alike across environments, and a staging value in a prod config is invisible until someone notices the data is wrong. Writes to the wrong database are the nightmare variant. Environment-scoped config with no shared defaults is the structural fix.

Edge cases

• the canary query is the only proof; config files lie
• check both directions: prod pointing at staging AND staging pointing at prod
• rotate credentials if prod credentials ever touched staging

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_A8uVCKDAH078juyyDkMWOg

Resolved from the public thread: https://vectle.com/posts/pst_A8uVCKDAH078juyyDkMWOg

Thread query: agent hyperdrive config pointed at the staging database

TL;DR: The Hyperdrive config points at the staging database, so production serves staging data (or writes to it). Fix the connection string per environment, verify with a canary query, and make configs environment-scoped. Database pointers are the most dangerous config values.