agent hallucinated a leaked key from a test fixture: trust error
Stops an agent from hallucinating a leaked key from a test fixture: verify against scanner output. Use when agents escalate fixtures to incidents. Not for real leaks.
TL;DR
Test fixtures look secret-shaped, and agents without verification escalate them. The rule: no incident without a scanner finding that names the file, line, and rule; fixtures get allowlisted, not incident-managed.
Error
agent hallucinated a leaked key from a test fixture: trust errorSteps
- Confirm the value is a fixture: it lives in test paths and matches known fake patterns. Expected: fixture proven.
- Check whether any scanner actually flagged it; if none did, there is no finding to act on. Expected: the absence of evidence noted.
- Allowlist the fixture so future runs skip it quietly. Expected: no repeat escalation.
- Require scanner citations for all leak claims in the agent's logic. Expected: claims need evidence.
- Review the agent's past escalations for other fixture false alarms. Expected: cleanup of noise.
When to use
- Agents escalating test fixtures as leaks.
- Tuning agent triage.
When not to use
- Scanner-confirmed findings (triage as leaks).
- Fixtures that are actually real keys (rotate immediately).
Tool compatibility
- Scanner findings; agent escalation logic.
Variant phrasings
agent flagged a test fixture as leak
Verify with the scanner.
false leak from test data
Allowlist it.
Why it happens
Agents see secret-shaped strings and skip the verification step that distinguishes fixtures from credentials.
Edge cases
- Some fixtures are real keys someone pasted as test data; treat ambiguous ones as real until proven otherwise.
- Fixture allowlists need the same review as other exceptions.
- Past false escalations erode trust; communicate the fix.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_tSch4dnevUVzoWwZTJ8vIA