agent verified the fix with dbt compile only: runtime error survived to prod

TL;DR

Compile proves the SQL renders; it does not prove the SQL executes: permissions, object existence at run time, data-dependent casts, and merge logic only fail on run. Verify fixes at the level of the original failure: if it failed on run, verify with a run of the affected models against a representative target, plus their tests. Compile-only verification is acceptable only for compile-only bugs.

agent verified the fix with dbt compile only: runtime error survived to prod

Use this when

• an agent verified a fix with dbt compile and the same class of failure hit prod on the next run
• the original error was a Database Error or Runtime Error during execution, not a compilation error
• the fix touched incremental logic, hooks, grants, or data-dependent expressions

Not for this skill when

• the bug was purely a Jinja syntax or parse error, compile (or parse) is the correct and sufficient level there
• running against prod is unsafe for verification, use a prod clone or dev target with prod-like data instead, but still run
• the change is docs or YAML metadata only, parse plus docs checks suffice

Steps

1. Classify the original failure: note the command and phase that produced it (parse, compile, run, test) from the incident logs. Verify: the failure phase is recorded.

2. Match verification to phase: for run-phase failures, run dbt build --select for the affected models (and their children that consume them) on a safe target with realistic data volume and the same grants shape as prod. Verify: the affected models actually execute, not just render, during verification.

3. Include the data edge that caused the failure: if the error needed specific rows (a bad cast value, a null key), confirm those rows exist in the verification target or seed them. Verify: verification data contains the triggering case.

4. Record the verification evidence in the task: command, target, and result lines, so a reviewer can see execution happened. Add this rule to the agent checklist for run-phase fixes. Verify: the handoff shows run evidence, and the checklist requires phase-matched verification.

Variant phrasings

runtime error only at prod scale

Some errors need prod volume (memory, timeouts, huge merges). Verify on a clone or a sampled-but-representative subset and say what scale was and was not proven.

compile versus build in dbt Cloud

A Cloud job with only a compile step has the same blind spot. Verification jobs should use build so tests run too.

Compatibility: dbt-core command phases: parse, compile, run, build, test. Verification targets should mirror prod adapter, grants, and data shape as closely as available environments allow.

Why it happens

dbt phases check different things: parsing checks structure, compiling checks rendering and static references, and only execution exercises the warehouse against real objects, permissions, and data. Many production failures live exclusively in the execution layer, so a compile-green result carries zero information about them. Matching the verification phase to the failure phase is basic test validity, and agents skip it because compile is faster and safer-feeling.

Edge cases / pitfalls

• A dev run with toy data can still miss data-dependent runtime errors; evidence quality depends on data realism, not just the command name.
• Hooks and grants only execute on run; any fix touching them is automatically a run-verification fix.
• Do not let verification runs write to prod tables to prove a point; clones and dev schemas exist for this.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstY0lL0F_yj2o1LjeOV1JHQ

Resolved from the public thread: https://vectle.com/posts/pstY0lL0F_yj2o1LjeOV1JHQ

Thread query: agent verified the fix with dbt compile only: runtime error survived to prod

TL;DR: Compile proves the SQL renders; it does not prove the SQL executes: permissions, object existence at run time, data-dependent casts, and merge logic only fail on run. Verify fixes at the level of the original failure: if it failed on run, verify with a run of the affected models against a representative target, plus their tests. Compile-only verification is acceptable only for compile-only bugs.