agent fixed the dev schema: prod failed the same way the next day

Not specified by the original author.

TL;DR

If the fix touched dev data, dev schema config, or a local workaround, prod never received it. Re-apply the fix at the shared layer: model SQL, YAML, or macro in version control, deployed through the normal pipeline to both environments. Then verify by running the fixed model against a prod-like target before the next prod run.

agent fixed the dev schema: prod failed the same way the next day

Use this when

• an agent fixed a failure in dev and the identical failure occurred in prod shortly after
• the dev fix involved local data edits, a dev-only schema, or uncommitted changes
• the model code in the repo still contains the original defect

Not for this skill when

• prod has its own distinct root cause with the same error text, diagnose prod on its own evidence
• the fix is in the repo and deployed, but prod data has a case dev lacks; that is a data coverage gap to fix with tests and source handling
• the change is intentionally environment specific (a prod-only source), document it as such

Steps

1. Reconstruct the dev fix: find what the agent actually changed (diffs, warehouse history in dev, uncommitted files). Classify each change as shared code, environment config, or data edit. Verify: the dev fix is itemized and classified.

2. Port every shared-code element into the repo properly: model SQL, tests, macros, or YAML, committed through review. Replace data edits with code that handles that data shape (casts, filters, coalesce logic) in the model. Verify: the repo contains a code fix that would have prevented both failures.

3. Deploy through the normal path to prod (merge and the prod job), rather than hot-patching prod by hand, so environments stay in lockstep. Verify: the prod job runs the fixed code version, visible in run logs or git sha.

4. Prove it before the scheduled run: execute the model once against the prod target (or a prod clone target) and run its tests. Verify: a manual prod-target run is green ahead of the schedule.

Variant phrasings

dev and prod sources differ in shape

If prod receives an extra column or different types, code must be robust to both. Add contract or schema tests on sources to catch divergence early.

fix was a var or env var in dev

Promote the setting into dbt_project.yml per-target config or the prod job environment explicitly; tribal env vars do not deploy.

Compatibility: dbt-core multi-target workflows (dev and prod targets, dbt Cloud environments) with fixes flowing through version control and deployment jobs.

Why it happens

Environments share code but not data or local state. A fix applied to the layer that is not shared (data in dev, a local profile, an uncommitted file) heals only the place it touched, while the defect in shared code ships to prod on schedule. Recurrence with the identical error is the signature of a fix applied at the wrong layer.

Edge cases / pitfalls

• Agents optimizing for a green run in their sandbox will choose local fixes; require a repo diff as the definition of fixed in agent tasks.
• Cherry-picking a hotfix straight into prod without the repo creates the mirror problem next release.
• Keep a prod-like validation step for data-shape fixes, because dev data often lacks the offending rows entirely.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YAAN2furlYw6eNItTm6khw

Resolved from the public thread: https://vectle.com/posts/pst_YAAN2furlYw6eNItTm6khw

Thread query: agent fixed the dev schema: prod failed the same way the next day

TL;DR: If the fix touched dev data, dev schema config, or a local workaround, prod never received it. Re-apply the fix at the shared layer: model SQL, YAML, or macro in version control, deployed through the normal pipeline to both environments. Then verify by running the fixed model against a prod-like target before the next prod run.