agent's fix passed tests locally but broke the production dag order: deploy error
Teaches a dbt agent that local test passes do not guarantee a safe deploy: CI must run the full DAG in a prod-like environment, and DAG order must be verified with dbt ls before merging. Use when a locally-green fix breaks production. Not for failures that also fail locally.
TL;DR
Local runs do not replicate production's DAG, data, or config. The agent must verify DAG order with dbt ls (checking the ref chain), run the full affected DAG in CI against a prod-like target, and only then deploy. A local-only green is not a deploy signal.
Error
agent's fix passed tests locally but broke the production dag order: deploy errorSteps
- Reproduce in CI: run the full DAG (or the affected subgraph with
+selectors) in the CI environment. Expected: the production failure reproduces. - Inspect the DAG order:
dbt ls --select [MODEL]+and compare against production's expectations. Expected: the ordering difference is visible. - Find the cause: a changed ref, a renamed model, or an environment-specific config that reorders execution. Expected: the specific divergence identified.
- Fix the ordering (correct the ref chain or the config) and rerun the full DAG in CI. Expected: green in the prod-like environment.
- Deploy only after CI is green on the full DAG. Expected: production matches the verified state.
When to use
- A fix passes locally but breaks production deployment or ordering.
- DAG-order-sensitive changes (renames, ref changes, new dependencies).
When not to use
- The failure reproduces locally (fix it locally first).
- The deploy failure is unrelated to the DAG (config, credentials, resources).
Tool compatibility
- dbt Core 1.0 and later; CI with a prod-like target.
Variant phrasings
Works on my machine, breaks the scheduled job
The scheduled job's target differs; CI must mirror it.
DAG order changed after a model rename
Renames change dependency edges; verify with dbt ls before merging.
Why it happens
Local dev targets differ from production in data volume, schema layout, and sometimes config. DAG order issues (a model built before its dependency is ready) only appear at production scale or with production's target configuration.
Edge cases
- Slim CI (state-based selection) must still cover the changed subgraph fully; too-slim selection misses ordering bugs.
- dbt Cloud jobs define their own commands; verify the job command matches what CI tested.
- Never deploy on a local-only verification for DAG-touching changes.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstRkHJ7epm5FufIcqZcCkDg