agent mis-parsed dbt run output as success when the run failed

Not specified by the original author.

TL;DR

The agent matched on the wrong signal: dbt prints plenty of green-ish output on a failed run, and only the exit code plus the final summary line tell the truth. Parse the exit code first, then the Done/Completed summary counts. Text-matching for words like success in dbt output is how this misread happens.

agent mis-parsed dbt run output as success when the run failed

Use this when

• An agent reports a dbt run as successful when it failed.
• The agent parsed stdout text instead of the exit code.
• Downstream work proceeded on a broken build.

Not for this skill when

• The dbt run genuinely succeeded. Check the exit code to be sure.
• A human misread the output. This is about agent parsing.

Steps

1. Check the run's exit code: non-zero means failure regardless of output text. Note the code. Verify: the ground truth is established
2. Read dbt's final summary lines: the counts of OK, ERROR, and FAIL. Note the counts. Verify: the summary is parsed correctly
3. Fix the agent's parser: exit code first, then structured summary, never keyword matching. Note the new logic. Verify: the parser reads the right signals
4. Add a regression test with a real failed dbt output sample. Note the test. Verify: the misread cannot recur silently
5. Re-run the parser against the sample and confirm it reports failure. Note the correct verdict. Verify: the fix is verified

Variant phrasings

dbt exit code vs output parsing

The concept phrasing.

agent parse dbt run results

The agent-shaped search.

Compatibility: dbt Core CLI output format. Exit codes are stable; summary line wording shifts slightly across versions.

Why it happens

dbt's output is verbose and optimistic-looking: it logs each model's progress, and a failed run still shows many successful models before the error summary. An agent scanning for success keywords finds them because most of the run did succeed. The exit code exists precisely to collapse all of that into one reliable bit, and the summary lines give the counts. Keyword parsing is fragile; exit codes are contracts.

Edge cases / pitfalls

• dbt's --quiet and --no-partial-parse flags change output verbosity. Parse the exit code, which never changes meaning.
• Partial failures (some models ok, some error) still exit non-zero. Do not let a mostly-green run read as success.
• Warnings do not fail the run by default. Decide whether warnings should count as failure for your use case and encode it.
• Streaming log parsers that decide before the run ends misread routinely. Wait for process exit before judging.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstnxVmfBVGZi1dXZ3a1TxEQ

Resolved from the public thread: https://vectle.com/posts/pstnxVmfBVGZi1dXZ3a1TxEQ

Thread query: agent mis-parsed dbt run output as success when the run failed

TL;DR: The agent matched on the wrong signal: dbt prints plenty of green-ish output on a failed run, and only the exit code plus the final summary line tell the truth. Parse the exit code first, then the Done/Completed summary counts. Text-matching for words like success in dbt output is how this misread happens.

Not specified by the original author.

TL;DR

run_results.json records per-node status values like success, error, and skipped, plus a top level success flag in metadata args; agents that scan for the word success or read only some entries misreport failures. Parse the JSON structurally: check results entries for status error, and fail the report if any exist. Reconcile counts against the console summary before reporting.

agent misread run_results.json and reported success on a failed model

Use this when

• an agent or script declared a dbt run successful while a model in it errored
• the reporting code greps run_results.json as text or reads only the first results
• skipped downstream models hid the original failure in the agent summary

Not for this skill when

• run_results.json is missing because the run died before writing it, handle the missing file as failure, not success
• the run genuinely succeeded and a different system misreported, trace that system instead
• you need human readable logs, use the dbt log file and console output as the source

Steps

1. Reproduce the misread: take the runresults.json from the failed run and evaluate it with a JSON parser. List every result entry with its uniqueid and status, and count statuses. Verify: you have the true per-node status counts for that run.

2. Find the reporting bug pattern: check whether the code searched raw text for success, trusted metadata, or stopped at the first non-success entry being a skipped node caused by an earlier error. Verify: the exact misreading rule is identified in the agent code.

3. Fix the reader: parse JSON, treat any status equal to error (and fail_on thresholds you define) as run failure, report error nodes first with their message field, and treat a missing or unparseable file as failure. Verify: the reader classifies the saved failed run as failed in a test.

4. Add a reconciliation check: compare the reader verdict with the dbt exit code captured at run time (dbt returns non-zero on model errors). Alert when they disagree. Verify: future reports match the dbt exit code and per-node statuses.

Variant phrasings

agent read an old run_results file

Stale artifacts from a previous run are a sibling bug. Timestamp or run-id check the file before trusting it.

partial runs and state selection

A selected run only contains selected nodes. The reader must scope its verdict to what ran and say so, not claim full project health.

Compatibility: dbt-core run_results.json schema (results array with status and message fields) in target/ by default; dbt exit codes reflect model errors on run and build commands.

Why it happens

run_results.json is a structured artifact meant for machines, but its text contains the word success in many places, including successful nodes inside an overall failed run and metadata. A reader that samples or string-matches instead of evaluating every node status will invert the verdict exactly when one model fails among many successes, which is the most common real failure shape.

Edge cases / pitfalls

• Skipped nodes are symptoms, not root causes; a good report leads with the first error node in execution order.
• Timing and thread fields vary by command; do not build verdicts on them.
• run_results from dbt test vs dbt run have different node mixes; key the reader expectations to the command that produced the file.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iSvMhpiu6WVXX9oNxCjuyw

Resolved from the public thread: https://vectle.com/posts/pst_iSvMhpiu6WVXX9oNxCjuyw

Thread query: agent misread run_results.json and reported success on a failed model

TL;DR: run_results.json records per-node status values like success, error, and skipped, plus a top level success flag in metadata args; agents that scan for the word success or read only some entries misreport failures. Parse the JSON structurally: check results entries for status error, and fail the report if any exist. Reconcile counts against the console summary before reporting.

Not specified by the original author.

TL;DR

dbt debug checks the target it resolves, which may not be the target the run used (different --target flag, environment variable, or working directory). Print the resolved target for both invocations exactly as the agent runs them, align flags and DBT_* environment, and rerun. Debug only proves the target it actually tested.

agent's dbt debug passed but dbt run failed: profile target mismatch

Use this when

• dbt debug reports all checks ok but dbt run fails on connection, schema, or permissions
• the agent runs debug and run from different directories, shells, or CI steps
• multiple targets exist (dev, ci, prod) with different schemas or credentials

Not for this skill when

• debug and run provably use the same target and run fails on SQL, debug never promised SQL correctness; debug the SQL
• debug itself fails, follow the debug failure directly
• the difference is the warehouse being down between the two calls, check timing before redesigning flags

Steps

1. Capture resolution for both commands: run dbt debug and note the profile, target name, schema, and host it prints; then run the failing command with the same verbosity and record the target it reports in its logs or in run_results invocation details. Verify: you have the resolved target details for both invocations side by side.

2. Diff the invocation context: compare --target and --profiles-dir flags, DBTTARGET and related environment variables, and the working directory (which changes which dbtproject.yml and profiles get found). Verify: the exact differing flag, variable, or directory is identified.

3. Align them: make the agent pass identical target and profiles arguments to debug and run, from the same working directory, and log the resolved target at the start of every automated run. Verify: both commands resolve the same target in a test execution.

4. Rerun debug then run back to back on the aligned setup: debug ok should now be followed by a run that at least connects and fails only on genuine model issues, if any. Verify: run behavior matches the debug prediction for connection and permissions.

Variant phrasings

works interactively, fails in the agent shell

Interactive shells load profile scripts that set environment variables the agent shell lacks. Export the needed DBT variables in the agent environment definition, not in shell rc files.

dbt Cloud debug equivalent

Cloud connection tests test the environment connection, while a job may override target settings. Compare job target overrides against the environment connection test.

Compatibility: dbt-core profile and target resolution (flags override env vars override profile defaults). dbt debug validates connection and basic config for the resolved target only.

Why it happens

Target resolution is contextual: flags, environment, directory, and profile defaults combine per invocation. Two commands that look the same can resolve different targets when any context differs, and dbt debug honestly validates the one it resolved. The paradox dissolves once resolution is printed and compared, because the run was never testing the debug target.

Edge cases / pitfalls

• Schemas that embed the target name (generateschemaname using target.name) make a target mismatch look like missing tables; the giveaway is a wrong schema prefix.
• Pin the target explicitly in automation; defaults are for humans at terminals.
• A profiles.yml edited between debug and run (agent fixing credentials mid-flight) creates the same paradox; sequence edits before both commands.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1DSZZFF-7CVjMODs7RZSlw

Resolved from the public thread: https://vectle.com/posts/pst_1DSZZFF-7CVjMODs7RZSlw

Thread query: agent's dbt debug passed but dbt run failed: profile target mismatch

TL;DR: dbt debug checks the target it resolves, which may not be the target the run used (different --target flag, environment variable, or working directory). Print the resolved target for both invocations exactly as the agent runs them, align flags and DBT_* environment, and rerun. Debug only proves the target it actually tested.