agent failed to distinguish the failing test from a similarly named passing one
Teaches an agent to disambiguate dbt tests with similar names by listing tests and selecting the exact node ID before running or fixing. Use when similarly named tests cause confusion. Not when test names are unique.
TL;DR
Two tests with near-identical names (for example on sibling models) are easy to confuse. The agent should list tests with dbt ls --resource-type test, pick the exact name matching the failure output, and always --select the full exact name when running or fixing.
Error
agent failed to distinguish the failing test from a similarly named passing oneSteps
- List the candidate tests:
dbt ls --resource-type test | grep -i [KEYWORD]. Expected: the full set of similarly named tests. - Compare against the failure output character by character (model name, column name, test type). Expected: the exact failing test identified.
- Run it alone with the exact name:
dbt test --select [EXACT TEST NAME]. Expected: the failure reproduces in isolation. - Fix the right test's underlying model or config. Expected: the change targets the failing case.
- Rerun the exact test, then the similarly named one, to confirm both are green. Expected: no confusion about which passed.
When to use
- Test names differ only by model or column and the agent mixed them up.
- Failure output names a test the agent cannot find.
When not to use
- Test names are unique and clear (no disambiguation needed).
- The agent copied the name from the failure output (already exact).
Tool compatibility
- dbt Core 1.0 and later.
dbt ls --resource-type testlists them all.
Variant phrasings
Agent fixed the passing twin of the failing test
The classic outcome; exact-name selection prevents it.
Generic tests auto-named identically on two models
not_null_orders_id vs not_null_orders_archive_id; read the full name.
Why it happens
dbt auto-generates test names from patterns, so sibling models produce sibling names. Agents skim the names and lock onto the wrong one, then "fix" a test that was already passing.
Edge cases
- Singular test filenames are custom; list
tests/directly for those. - Copy the test name from the failure output rather than retyping it.
- When in doubt, run both twins; the failure output tells you which is which.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstwUyWLGUQomQf5ebKb8JjA