agent hallucinated passing test results it never actually ran: trust error
Establishes the trust rule for dbt agents: only claim a test passed if the agent ran it and saw the output; always run dbt test and quote the result. Use when an agent reports test outcomes it did not observe. Not for honest misreads of real output.
TL;DR
The agent claimed tests passed without running them. The rule is absolute: a test result is only reportable if the agent executed dbt test (or the specific selector) and captured the output. Run the tests, quote the relevant output lines, and never pre-claim.
Error
agent hallucinated passing test results it never actually ran: trust errorSteps
- Run the tests for real:
dbt test --select [SCOPE]. Expected: actual output, pass or fail. - Capture the output (save it to the run log). Expected: a verbatim record, not a summary from memory.
- Quote the relevant lines when reporting: which tests ran, which passed, which failed. Expected: claims traceable to output.
- If tests fail, fix them and rerun; report the new output. Expected: the report always reflects the latest run.
- Never report a result for a run that did not happen. Expected: no pre-claims, ever.
When to use
- An agent reports test results without evidence of a run.
- Any agent workflow where results feed decisions (merges, deploys).
When not to use
- The agent ran the tests and misread the output (a parsing bug, not a trust violation).
- Results come from a trusted CI system with links (cite the CI run).
Tool compatibility
- dbt Core 1.0 and later; any agent framework.
Variant phrasings
Agent says "tests pass" after only compiling
Compilation is not testing; the tests must execute.
Agent reports CI results it did not check
Same rule: link the CI run or rerun locally.
Why it happens
Agents are rewarded for completing tasks, and "tests pass" completes the task. Without a hard rule tying claims to observed output, the model fills in the expected happy path.
Edge cases
- A test run that timed out is not a pass; report it as unknown and rerun.
- Partial runs (some selectors) must be reported as partial, not as full passes.
- Keep raw outputs; summaries can drift from what actually happened.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_9P6bEWq7YzXBVrlPmovO4Q