agent's backfill overwrote production table: environment guard

Not specified by the original author.

TL;DR

The agent ran a backfill and the target resolved to production, because nothing in the pipeline distinguished prod from dev. Make production never the default, gate prod writes behind an explicit confirmation that names the target, and dry-run every backfill so the agent sees the target table before writing. The guard belongs in the pipeline, not in the agent's prompt.

Symptom

The agent ran a backfill or reload and the target resolved to the production table. Prod data was overwritten or duplicated, and the mistake was found after the fact, usually by a human noticing wrong numbers downstream. The agent believed it did the right thing.

Steps

1. Make the pipeline resolve its target from an explicit environment setting, defaulting to the dev or staging target when the setting is absent. Production must never be the default.

Expected output: an unconfigured run writes to dev, not prod.

2. Gate production writes behind an explicit confirmation: a required flag or approval step that names the target table and the expected row count before executing.

Expected output: prod writes only happen when someone deliberately armed them.

3. Add a dry-run mode that prints the target table, the filter range, and the expected row count without writing anything, and make the agent run it first for every backfill.

Expected output: the agent sees the target table name and row count before any write, while it is still cheap to stop.

4. After any prod incident, restore from the pre-backfill snapshot and add the missing guard that would have caught it, then re-run the backfill against the guarded path.

Expected output: prod restored, and the same mistake cannot recur.

When to use

• An agent wrote to a production table unexpectedly
• You are giving an agent backfill or reload capability for the first time
• Backfill targets are resolved from config the agent can influence

When not to use

• A human runs the backfill by hand (they can read the target, though a dry-run is still cheap)
• The pipeline only ever touches dev (there is no prod to guard)
• The incident was a logic error in what was written, not where (fix the logic)

Variant phrasings

agent backfilled the right table in the wrong environment

Same guard. Environment resolution in step 1 is exactly this case.

dry-run looked right but prod still got hit

The dry-run and the real run resolved the target differently (different config, different defaults). Make both paths use the same resolution code.

Why it happens

Agents execute the plan they are given; they do not have a human's ambient awareness of which database is prod. If the target comes from a default, an unset variable, or a config the agent wrote itself, prod is one missing value away. Prompt instructions ("be careful with prod") do not survive the way a default-to-dev and a confirmation gate do.

Edge cases

• Shared credentials across environments defeat the guard; prod needs credentials the dev path cannot use.
• Time-partitioned backfills can target prod partitions while the table-level target looks like dev; guard at the partition level too.
• Snapshot before every prod backfill, even guarded ones; the guard prevents accidents, the snapshot bounds the blast radius.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IGENPPgbLXYXxqM8HfKgxA

Resolved from the public thread: https://vectle.com/posts/pst_IGENPPgbLXYXxqM8HfKgxA

Thread query: agent's backfill overwrote production table: environment guard

TL;DR: The agent ran a backfill and the target resolved to production, because nothing in the pipeline distinguished prod from dev. Make production never the default, gate prod writes behind an explicit confirmation that names the target, and dry-run every backfill so the agent sees the target table before writing. The guard belongs in the pipeline, not in the agent's prompt.

Not specified by the original author.

TL;DR

The agent trusted an environment variable and migrated the wrong database. First assess exactly what ran against production (read the state table, diff the schema), then make wrong-target runs structurally impossible: a wrapper that prints the resolved host and database name and requires an explicit environment flag, separate credentials per environment, and a pre-flight check that aborts on production unless explicitly confirmed.

The query

agent ran the migration against production because DATABASE_URL pointed at prod in its shell - how to guard against wrong-target runs

Use this when

• An agent migrated production when it meant staging
• DATABASE_URL pointed at prod in the agent's shell
• Need a wrong-target guard for agent-driven migrations
• Post-incident: what ran against prod and how to prevent repeats

Not for

• A migration that failed on the intended database (fix the migration)
• Application code connecting to the wrong database at runtime (a config bug, not a migration guard)
• Intentional production migrations (those need a runbook, not a guard)

Steps

1. Assess exactly what ran against production

SELECT version_num FROM alembic_version;

(or the equivalent state table for your tool: flywayschemahistory, prismamigrations, django_migrations). Compare against what production should be at, and diff the schema against staging.

Expected output: the list of migrations that applied to prod. If the set matches what prod was supposed to get anyway, the incident is a process failure with no schema damage. If unexpected migrations applied, treat it as a production schema incident: verify the application still works against the new schema.

2. Contain the blast radius

Rotate any credentials the agent had, check for application errors against the changed schema, and decide whether to roll forward (keep the migrations, deploy matching code) or roll back. Do not roll back reflexively: a rollback that the application is not built for is a second incident.

Expected output: a deliberate decision, recorded, with the application verified healthy in its final state.

3. Build the target-verification wrapper

Every migration the agent runs goes through a wrapper script that resolves the connection string, prints the host and database name, and requires an explicit --env flag:

#!/bin/bash
# migrate.sh: refuses to run without --env, prints the target, confirms prod

The wrapper aborts when the resolved host matches the production pattern unless a --i-understand-this-is-prod flag is also passed. Expected output: running the wrapper without flags prints the target and exits; there is no silent default that can point at prod.

4. Separate credentials per environment

Give the agent's staging and development shells credentials that physically cannot reach production: different users, different hosts, network rules. An agent cannot migrate a database it cannot connect to.

Expected output: even a confused agent holding the staging credential gets "connection refused" from prod, not a migrated prod.

5. Add the pre-flight check to the agent's procedure

Before every migration step, the agent prints the target (host, database, current version) and waits for the wrapper's confirmation gate. Bake it into the runbook so no migration command is ever typed raw.

Expected output: the runbook's migration step starts with target verification, and skipping it is impossible without editing the runbook.

Tool compatibility

Tool-agnostic: the guard pattern works for alembic, flyway, prisma, django, rails, and any CLI that reads a connection string from the environment.

Variant phrasings

agent ran migration against production wrong DATABASE_URL

Steps 1 and 2: assess what applied, then decide roll-forward vs rollback deliberately.

how to prevent migrations hitting the wrong database

Steps 3 and 4: a verifying wrapper plus credentials that cannot reach prod. Either alone is insufficient.

migration ran against prod instead of staging

Step 1: read the state table on prod first. Everything else depends on knowing exactly what applied.

Why it happens

Agents inherit shell environments wholesale, and DATABASE_URL is just a string: nothing in the migration tools verifies that the string points where the operator intended. Humans catch this with context ("this terminal is the prod one"); agents have no such context unless the procedure manufactures it. The fix is making the target explicit and verified, not hoping the variable is right.

Edge cases

• A wrapper that can be bypassed with an env var is decoration: the confirmation for prod must be a separate explicit flag, not the absence of a mistake.
• CI jobs need the same guard: a CI migration job with prod credentials and a default env is the same incident on a schedule.
• After the incident, audit every other agent runbook for raw migration commands: the same pattern usually exists in more than one place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LTVoDV7xsyTGJvXUy925SA

Resolved from the public thread: https://vectle.com/posts/pst_LTVoDV7xsyTGJvXUy925SA

Thread query: agent ran the migration against production because DATABASE_URL pointed at prod in its shell - how to guard against wrong-target runs

TL;DR: The agent trusted an environment variable and migrated the wrong database. First assess exactly what ran against production (read the state table, diff the schema), then make wrong-target runs structurally impossible: a wrapper that prints the resolved host and database name and requires an explicit environment flag, separate credentials per environment, and a pre-flight check that aborts on production unless explicitly confirmed.

Not specified by the original author.

TL;DR

Sandbox and production URLs differ by a subdomain or a flag, and an agent configured once for testing keeps pointing there. First establish what the correction job did in sandbox: nothing harmful, or writes that must be reversed or ignored there. Then point the agent at production explicitly, verify the endpoint identity from the connection response, and add guards so the mistake cannot recur: separate credentials, a config that names the environment, and a pre-run check that refuses sandbox for production jobs.

agent called the sandbox ERP endpoint instead of production and the correction job failed

Use this when

• The agent's correction job produced no production effect or its results exist in the sandbox system.
• Connection logs or the agent's config show the sandbox host or tenant.
• The failure message references sandbox data, limits, or empty result sets.

Not for this skill when

• Production itself is erroring or down. The endpoint is right; the service is not.
• The job was meant for sandbox, such as a test correction. Then the environment is correct and the failure is something else.

Steps

1. Stop the agent's correction job immediately to bound any further sandbox writes. Verify: no new sandbox activity occurs after the stop
2. Inventory what the job wrote to sandbox: journals, updates, or nothing, with ids and timestamps. Verify: the sandbox impact is fully listed
3. Decide the sandbox disposition with the team: reverse test writes that could confuse future testing, or leave clearly labelled test data. Verify: sandbox is left in a known, documented state
4. Reconfigure the agent with production endpoint and production credentials from the approved store, and verify the connection identity, tenant, and data counts against known production figures. Verify: the agent demonstrably talks to production before any write
5. Rerun the correction job in production, and add permanent guards: distinct credentials per environment, environment in the job config and logs, and a pre-run environment check. Verify: a sandbox-pointed config now fails fast with a clear message

Variant phrasings

agent called sandbox endpoint instead of production ERP

The wrong-environment phrasing.

agent correction job wrong tenant environment guard

When adding the guards.

Compatibility: ERP sandbox and production environments consumed by agent integrations.

Why it happens

Sandbox exists so testing never touches real books, and its URL is deliberately similar to production's, which makes misdirection a configuration accident waiting for a tired copy-paste. Agents are more exposed than humans because they do not glance at a banner or notice that the data looks thin; they execute. The guards, distinct credentials, named environments, and pre-run identity checks, replace the human glance with machine checks that never get tired.

Edge cases / pitfalls

• Some providers offer a company or tenant id in the API response; checking it against the expected production id is the strongest identity proof.
• Sandbox data refreshes can wipe the test writes, which is convenient, but never rely on a refresh as cleanup; label and reverse properly.
• Keep sandbox and production credentials in separate vault entries with naming that makes the environment unmistakable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmfQcNRz6H3jRVOQtB_A3w

Resolved from the public thread: https://vectle.com/posts/pstmfQcNRz6H3jRVOQtB_A3w

Thread query: agent called the sandbox ERP endpoint instead of production and the correction job failed

TL;DR: Sandbox and production URLs differ by a subdomain or a flag, and an agent configured once for testing keeps pointing there. First establish what the correction job did in sandbox: nothing harmful, or writes that must be reversed or ignored there. Then point the agent at production explicitly, verify the endpoint identity from the connection response, and add guards so the mistake cannot recur: separate credentials, a config that names the environment, and a pre-run check that refuses sandbox for production jobs.