agent hallucinating data quality rules failure invented non-existent constraints error

TL;DR

Only create rules you can point to in a source: derive each constraint from the schema definition, a data contract, existing tests, or a profiling result, and record that source next to the rule. An invented constraint, such as a uniqueness or range rule nobody agreed, fails good data or blocks a pipeline on a requirement that never existed.

agent hallucinating data quality rules failure invented non-existent constraints error

Use this when

• The check or run fails with this symptom: the agent invents data quality constraints that were never specified.
• You are working on an agent proposing or generating data quality rules for a table and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A rule that exists in a contract the agent has not read yet, where the fix is reading the contract rather than deleting the rule.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List every rule the agent created and require a source for each: contract clause, schema constraint, existing test, or profiling evidence. Verify: each rule has a named source or is flagged unsourced
2. Delete or disable unsourced rules instead of tuning their thresholds to pass. Verify: no active rule lacks a source
3. For profiling-derived rules, attach the observed values, such as the actual min, max, or distinct count, that justify the bound. Verify: each derived rule cites the profile numbers behind it
4. Run the surviving rules and confirm failures now correspond to real contract or schema violations. Verify: remaining failures trace to documented expectations

Variant phrasings

agent invented data quality rules not in contract

The invented-rule phrasing.

hallucinated constraints failing valid data

The false-failure phrasing.

agent added checks nobody specified pipeline blocked

The blocked-pipeline phrasing.

Compatibility: Agents authoring data quality rules for dbt tests, Soda, or Great Expectations suites

Why it happens

Agents asked to add checks will produce a plausible standard rule set, uniqueness, ranges, allowed values, whether or not the business ever agreed to those constraints. The rules look professional, so they get merged, and only fail later against legitimate data that never promised to satisfy them.

Edge cases / pitfalls

• Profiling a biased sample produces wrong bounds; profile a full recent partition before deriving ranges.
• Keep generated rules in a separate file or suite section so reviewers can see exactly what the agent added.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstx1BjLshRkT8UYQyFYqMIg