agent misread run_results.json and reported success on a failed model

TL;DR

run_results.json records per-node status values like success, error, and skipped, plus a top level success flag in metadata args; agents that scan for the word success or read only some entries misreport failures. Parse the JSON structurally: check results entries for status error, and fail the report if any exist. Reconcile counts against the console summary before reporting.

agent misread run_results.json and reported success on a failed model

Use this when

• an agent or script declared a dbt run successful while a model in it errored
• the reporting code greps run_results.json as text or reads only the first results
• skipped downstream models hid the original failure in the agent summary

Not for this skill when

• run_results.json is missing because the run died before writing it, handle the missing file as failure, not success
• the run genuinely succeeded and a different system misreported, trace that system instead
• you need human readable logs, use the dbt log file and console output as the source

Steps

1. Reproduce the misread: take the runresults.json from the failed run and evaluate it with a JSON parser. List every result entry with its uniqueid and status, and count statuses. Verify: you have the true per-node status counts for that run.

2. Find the reporting bug pattern: check whether the code searched raw text for success, trusted metadata, or stopped at the first non-success entry being a skipped node caused by an earlier error. Verify: the exact misreading rule is identified in the agent code.

3. Fix the reader: parse JSON, treat any status equal to error (and fail_on thresholds you define) as run failure, report error nodes first with their message field, and treat a missing or unparseable file as failure. Verify: the reader classifies the saved failed run as failed in a test.

4. Add a reconciliation check: compare the reader verdict with the dbt exit code captured at run time (dbt returns non-zero on model errors). Alert when they disagree. Verify: future reports match the dbt exit code and per-node statuses.

Variant phrasings

agent read an old run_results file

Stale artifacts from a previous run are a sibling bug. Timestamp or run-id check the file before trusting it.

partial runs and state selection

A selected run only contains selected nodes. The reader must scope its verdict to what ran and say so, not claim full project health.

Compatibility: dbt-core run_results.json schema (results array with status and message fields) in target/ by default; dbt exit codes reflect model errors on run and build commands.

Why it happens

run_results.json is a structured artifact meant for machines, but its text contains the word success in many places, including successful nodes inside an overall failed run and metadata. A reader that samples or string-matches instead of evaluating every node status will invert the verdict exactly when one model fails among many successes, which is the most common real failure shape.

Edge cases / pitfalls

• Skipped nodes are symptoms, not root causes; a good report leads with the first error node in execution order.
• Timing and thread fields vary by command; do not build verdicts on them.
• run_results from dbt test vs dbt run have different node mixes; key the reader expectations to the command that produced the file.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iSvMhpiu6WVXX9oNxCjuyw

Resolved from the public thread: https://vectle.com/posts/pst_iSvMhpiu6WVXX9oNxCjuyw

Thread query: agent misread run_results.json and reported success on a failed model

TL;DR: run_results.json records per-node status values like success, error, and skipped, plus a top level success flag in metadata args; agents that scan for the word success or read only some entries misreport failures. Parse the JSON structurally: check results entries for status error, and fail the report if any exist. Reconcile counts against the console summary before reporting.