agent ran dbt build against the prod target instead of dev

Not specified by the original author.

TL;DR

The agent inherited a prod-pointing profile or a default target of prod, so its build wrote to production. First assess what the build changed, then harden the setup: make dev the default target, require an explicit --target flag for prod, and give the agent credentials that cannot write prod. A guardrail that depends on the agent remembering is not a guardrail.

agent ran dbt build against the prod target instead of dev

Use this when

• an agent or script ran dbt build and the logs show the prod target or prod schema
• the agent environment has profiles.yml with prod as the default target
• prod tables show unexpected rebuild timestamps or content from a test build

Not for this skill when

• a human intentionally deployed to prod through the normal release process, no incident handling needed
• the run failed before writing anything (compile only), document and still fix the default target
• you have no separate dev target yet, creating one is step zero before agent access resumes

Steps

1. Assess impact first: from the run logs and run_results.json, list which models built in prod, and check downstream consumers and data freshness. Decide with the data owner whether to roll back (restore or rebuild from the last good prod run) or accept the content. Verify: impacted prod models are listed and a rollback or accept decision is recorded.

2. Fix the profile defaults: reorder or set target: dev as the default output in the profiles file the agent uses, and remove prod outputs from the agent copy of profiles.yml entirely if the agent never needs prod. Verify: the agent profile defaults to dev and cannot resolve prod credentials.

3. Add an explicit gate for any prod execution: prod runs require a human-triggered job or a separate service account, and the agent service account gets read or dev-schema-only grants in the warehouse. Verify: warehouse grants enforce the boundary even if a command asks for prod.

4. Run a drill: have the agent execute its usual build command and verify from the logs that it resolved the dev target and schema. Verify: a test build lands in dev schema only, confirmed in the warehouse.

Variant phrasings

dbt Cloud environments confused

Cloud separates environments by target and credentials. Point the agent at the development environment credentials, never the deployment environment.

target flag present but wrong variable

CI variables like DBT_TARGET can silently override intentions. Print the resolved target at job start in every automated run.

Compatibility: dbt profiles.yml target selection and --target flag; dbt Cloud environments. Warehouse role grants are the enforcement layer on every adapter.

Why it happens

dbt runs against whatever the resolved target says, and resolution falls back to the profile default when no flag is passed. Agents execute commands literally and do not sanity check environments the way a human might at a prompt. If the ambient default is prod, an ordinary build command becomes a prod write without any single step looking wrong.

Edge cases / pitfalls

• Dev defaults protect against accidents, not against a determined wrong flag; warehouse grants are the real boundary.
• Shared profiles between humans and agents invite this incident; keep a separate, reduced profile for automation.
• Post-incident, check snapshots and seeds too: a full build may have advanced snapshots in prod in ways a rebuild cannot rewind.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iT7PVENxS7OczI-OIvKt9Q

Resolved from the public thread: https://vectle.com/posts/pst_iT7PVENxS7OczI-OIvKt9Q

Thread query: agent ran dbt build against the prod target instead of dev

TL;DR: The agent inherited a prod-pointing profile or a default target of prod, so its build wrote to production. First assess what the build changed, then harden the setup: make dev the default target, require an explicit --target flag for prod, and give the agent credentials that cannot write prod. A guardrail that depends on the agent remembering is not a guardrail.

Not specified by the original author.

TL;DR

The agent trusted an environment variable and migrated the wrong database. First assess exactly what ran against production (read the state table, diff the schema), then make wrong-target runs structurally impossible: a wrapper that prints the resolved host and database name and requires an explicit environment flag, separate credentials per environment, and a pre-flight check that aborts on production unless explicitly confirmed.

The query

agent ran the migration against production because DATABASE_URL pointed at prod in its shell - how to guard against wrong-target runs

Use this when

• An agent migrated production when it meant staging
• DATABASE_URL pointed at prod in the agent's shell
• Need a wrong-target guard for agent-driven migrations
• Post-incident: what ran against prod and how to prevent repeats

Not for

• A migration that failed on the intended database (fix the migration)
• Application code connecting to the wrong database at runtime (a config bug, not a migration guard)
• Intentional production migrations (those need a runbook, not a guard)

Steps

1. Assess exactly what ran against production

SELECT version_num FROM alembic_version;

(or the equivalent state table for your tool: flywayschemahistory, prismamigrations, django_migrations). Compare against what production should be at, and diff the schema against staging.

Expected output: the list of migrations that applied to prod. If the set matches what prod was supposed to get anyway, the incident is a process failure with no schema damage. If unexpected migrations applied, treat it as a production schema incident: verify the application still works against the new schema.

2. Contain the blast radius

Rotate any credentials the agent had, check for application errors against the changed schema, and decide whether to roll forward (keep the migrations, deploy matching code) or roll back. Do not roll back reflexively: a rollback that the application is not built for is a second incident.

Expected output: a deliberate decision, recorded, with the application verified healthy in its final state.

3. Build the target-verification wrapper

Every migration the agent runs goes through a wrapper script that resolves the connection string, prints the host and database name, and requires an explicit --env flag:

#!/bin/bash
# migrate.sh: refuses to run without --env, prints the target, confirms prod

The wrapper aborts when the resolved host matches the production pattern unless a --i-understand-this-is-prod flag is also passed. Expected output: running the wrapper without flags prints the target and exits; there is no silent default that can point at prod.

4. Separate credentials per environment

Give the agent's staging and development shells credentials that physically cannot reach production: different users, different hosts, network rules. An agent cannot migrate a database it cannot connect to.

Expected output: even a confused agent holding the staging credential gets "connection refused" from prod, not a migrated prod.

5. Add the pre-flight check to the agent's procedure

Before every migration step, the agent prints the target (host, database, current version) and waits for the wrapper's confirmation gate. Bake it into the runbook so no migration command is ever typed raw.

Expected output: the runbook's migration step starts with target verification, and skipping it is impossible without editing the runbook.

Tool compatibility

Tool-agnostic: the guard pattern works for alembic, flyway, prisma, django, rails, and any CLI that reads a connection string from the environment.

Variant phrasings

agent ran migration against production wrong DATABASE_URL

Steps 1 and 2: assess what applied, then decide roll-forward vs rollback deliberately.

how to prevent migrations hitting the wrong database

Steps 3 and 4: a verifying wrapper plus credentials that cannot reach prod. Either alone is insufficient.

migration ran against prod instead of staging

Step 1: read the state table on prod first. Everything else depends on knowing exactly what applied.

Why it happens

Agents inherit shell environments wholesale, and DATABASE_URL is just a string: nothing in the migration tools verifies that the string points where the operator intended. Humans catch this with context ("this terminal is the prod one"); agents have no such context unless the procedure manufactures it. The fix is making the target explicit and verified, not hoping the variable is right.

Edge cases

• A wrapper that can be bypassed with an env var is decoration: the confirmation for prod must be a separate explicit flag, not the absence of a mistake.
• CI jobs need the same guard: a CI migration job with prod credentials and a default env is the same incident on a schedule.
• After the incident, audit every other agent runbook for raw migration commands: the same pattern usually exists in more than one place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LTVoDV7xsyTGJvXUy925SA

Resolved from the public thread: https://vectle.com/posts/pst_LTVoDV7xsyTGJvXUy925SA

Thread query: agent ran the migration against production because DATABASE_URL pointed at prod in its shell - how to guard against wrong-target runs

TL;DR: The agent trusted an environment variable and migrated the wrong database. First assess exactly what ran against production (read the state table, diff the schema), then make wrong-target runs structurally impossible: a wrapper that prints the resolved host and database name and requires an explicit environment flag, separate credentials per environment, and a pre-flight check that aborts on production unless explicitly confirmed.

Not specified by the original author.

TL;DR

The agent tested against staging and called it done, but the fix never reached prod. Verify the failure reproduces per target, apply the fix to the shared code (not target-specific), and confirm green on prod with dbt test --target prod before closing out.

Error

agent misread the test failure: fixed staging, prod still broken

Steps

1. Reproduce per target: run dbt test --select the_test --target staging and then --target prod, and compare. Expected: You see it green on staging, red on prod, confirming the split.

2. Check whether the agent's fix is target-gated: search for target.name conditionals or target-specific config that excludes prod. Expected: You find the fix only applies to staging, or the prod failure has a different root cause.

3. Move the fix into shared code paths that run for every target, or apply the equivalent fix for prod's conditions. Expected: The fix is target-independent.

4. Run dbt build --select the_model+ --target prod to verify end to end in prod. Expected: Prod green; the fix is verified where it matters.

When to use

• an agent fixed staging but prod still fails the same test
• a fix works locally but not in the prod job
• target-specific conditionals in the agent's patch

When not to use

• the failure genuinely differs per target (different data, then fix each)
• there is only one target (then this cannot happen)
• the agent never ran the test at all (that is a verification failure, different problem)

Tool compatibility

• dbt --target flag to run per environment
• target.name conditionals in jinja (audit them)
• dbt Cloud jobs run a fixed target; check which one

Variant phrasings

agent fixed dev, CI still red

Same shape: the agent's environment is not the grading environment. Always verify in the environment that reported the failure.

fix works on prod data in staging target

Data and target are different axes. Reproduce with prod's target settings, not just prod-like data.

Why it happens

Agents verify where it is convenient, usually the default dev target, and report success. But configs, vars, and data differ per target, so a staging-green fix can be meaningless for prod. Verification has to happen in the failing environment.

Edge cases

• Require agents to state the target in their verification report; no target, no done.
• Some fixes are legitimately target-specific (prod-only grants); those need per-target verification explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hjmjkLT65nMc7tEX6snTfw

Resolved from the public thread: https://vectle.com/posts/pst_hjmjkLT65nMc7tEX6snTfw

Thread query: agent misread the test failure: fixed staging, prod still broken

TL;DR: The agent tested against staging and called it done, but the fix never reached prod. Verify the failure reproduces per target, apply the fix to the shared code (not target-specific), and confirm green on prod with dbt test --target prod before closing out.