agent hallucinated a dbt test name that doesn't exist: verification error
Teaches a dbt agent to verify test names against dbt ls before acting, so it never operates on a hallucinated name. Use when an agent references a test that does not exist. Not for tests that exist but are misconfigured.
TL;DR
The agent acted on a test name it invented. The rule is simple: never trust a recalled test name; always verify with dbt ls --select [NAME] or list tests with dbt ls --resource-type test first. If the name does not exist, find the real one before doing anything.
Error
agent hallucinated a dbt test name that doesn't exist: verification errorSteps
- Stop and verify:
dbt ls --select [CLAIMED NAME]. Expected: either the test exists (proceed) or it does not (the hallucination is confirmed). - If it does not exist, list the real tests:
dbt ls --resource-type test | grep -i [KEYWORD]. Expected: the actual test names matching the intent. - Match the intent to a real test name and confirm it with one more
dbt ls. Expected: a verified, exact name. - Log the correction (claimed vs actual) so the mistake is visible. Expected: the record shows what happened.
- Proceed using only the verified name. Expected: no further hallucinated references.
When to use
- An agent names a dbt test that turns out not to exist.
- Any agent workflow that recalls resource names from memory.
When not to use
- The test exists but fails (a real test problem, not a hallucination).
- The agent quoted the name from actual command output (that is verification already).
Tool compatibility
- dbt Core 1.0 and later.
dbt lsis the source of truth for names.
Variant phrasings
Agent ran dbt test --select with a wrong name
dbt test with no matches is the symptom; verification is the prevention.
Agent edited a test file that does not exist
Same rule applied to files: list before editing.
Why it happens
Language models complete patterns plausibly, and dbt test names follow predictable patterns (not_null_[MODEL]_[COLUMN]). Plausible is not the same as real, and acting on a plausible name wastes runs or edits the wrong thing.
Edge cases
- Test names include the package and test type; a partial name may match nothing even when the test exists.
- Custom test names (singular tests) are the easiest to hallucinate; always list
tests/too. - Make verification a habit for every name the agent did not copy from tool output.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_G-XMLRBBhyhRpXZ1LNOAwQ