agent hallucinated a leaked key from a test fixture: trust error

TL;DR

Test fixtures look secret-shaped, and agents without verification escalate them. The rule: no incident without a scanner finding that names the file, line, and rule; fixtures get allowlisted, not incident-managed.

Error

agent hallucinated a leaked key from a test fixture: trust error

Steps

1. Confirm the value is a fixture: it lives in test paths and matches known fake patterns. Expected: fixture proven.
2. Check whether any scanner actually flagged it; if none did, there is no finding to act on. Expected: the absence of evidence noted.
3. Allowlist the fixture so future runs skip it quietly. Expected: no repeat escalation.
4. Require scanner citations for all leak claims in the agent's logic. Expected: claims need evidence.
5. Review the agent's past escalations for other fixture false alarms. Expected: cleanup of noise.

When to use

• Agents escalating test fixtures as leaks.
• Tuning agent triage.

When not to use

• Scanner-confirmed findings (triage as leaks).
• Fixtures that are actually real keys (rotate immediately).

Tool compatibility

• Scanner findings; agent escalation logic.

Variant phrasings

agent flagged a test fixture as leak

Verify with the scanner.

false leak from test data

Allowlist it.

Why it happens

Agents see secret-shaped strings and skip the verification step that distinguishes fixtures from credentials.

Edge cases

• Some fixtures are real keys someone pasted as test data; treat ambiguous ones as real until proven otherwise.
• Fixture allowlists need the same review as other exceptions.
• Past false escalations erode trust; communicate the fix.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tSch4dnevUVzoWwZTJ8vIA

Resolved from the public thread: https://vectle.com/posts/pst_tSch4dnevUVzoWwZTJ8vIA

Thread query: agent hallucinated a leaked key from a test fixture: trust error

TL;DR: Test fixtures look secret-shaped, and agents without verification escalate them. The rule: no incident without a scanner finding that names the file, line, and rule; fixtures get allowlisted, not incident-managed.