agent added a not_null test that fails on legitimate source nulls

TL;DR

The agent assumed the column should never be null, but the source legitimately sends nulls. Scope the test to where nulls are truly invalid (a where clause on the test), or drop the test and document why nulls are fine. Do not fix the data to satisfy the test.

Error

agent added a not_null test that fails on legitimate source nulls

Steps

1. Look at the failing rows: are the nulls legitimate business states (e.g. cancelled orders with no ship date)? Ask the data owner if unsure. Expected: You can say which nulls are valid and which would be real bugs.

2. If some nulls are valid, scope the test: add a where clause excluding the legitimate cases, e.g. where: "status != 'cancelled'". Expected: The test config narrows to the truly-invalid population.

3. If nulls are broadly legitimate, remove the test and leave a comment in schema.yml explaining why. Expected: No test asserts something the business does not believe.

4. Run dbt test --select the_test to confirm green. Expected: Test passes on the scoped population.

When to use

• an agent-added not_null test fails on nulls the business accepts
• a generic test was applied without checking the data first
• the test fails right after the agent created it

When not to use

• the nulls are genuine data bugs (then fix the source or the model)
• the test was human-written and deliberate (talk to the author)
• the column should never be null per the contract (enforce it, do not scope it away)

Tool compatibility

• dbt generic tests; where config scopes the tested rows
• schema.yml comments document why a test was removed
• dbt test --select runs the single test

Variant phrasings

agent added unique test failing on legit dupes

Same pattern: the agent assumed uniqueness the business does not guarantee. Scope with where or remove.

agent added accepted_values failing on new valid values

The source added a legitimate new value. Update the values list, do not filter the data.

Why it happens

Agents pattern-match test names to columns without learning the business rules. not_null on every id-looking column feels safe, but real sources have legitimate nulls. The test encodes an assumption nobody validated.

Edge cases

• A where clause that excludes too much makes the test vacuous; keep the exclusion tight and documented.
• If nulls are valid today but should shrink over time, use warn severity instead of removing the test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XK9zrDuEJJw8l7zuM4Ag0g