eval agent hallucinating test failures not in the log: how to ground it

TL;DR

An eval agent that summarizes test results from a long log will sometimes invent failures, names and all. Ground every claim: require one verbatim quoted log line per reported failure, cross-check test names against the runner's collected list, and add an automated diff between the agent's claim list and the parsed JUnit or TAP output. Claims without quotes get rejected.

Error

[agent summary] 3 failures: test_login_expired, test_checkout_coupon, test_refund_partial
[pytest log tail]  212 passed in 41.2s
The named tests do not appear anywhere in the log.

Steps

1. Make the agent re-read the raw log and output the exact failure lines verbatim, with line numbers. Expected: real failures come back as quotes; invented ones cannot be quoted and the claim collapses.
2. Verify each named test exists: pytest --collect-only -q and search the list. Expected: every claimed test name appears in the collected list, or the claim is dropped.
3. Enforce a summary format: one quoted log line per claimed failure, no quote, no claim. Expected: the agent's failure list shrinks to what it can evidence.
4. Feed logs in numbered chunks instead of one truncated paste so citations point at real line numbers. Expected: every citation resolves to an actual log line.
5. Add a machine check: parse the JUnit XML and diff its failure list against the agent's list. Expected: mismatch count is zero before the summary is accepted.

When to use

• The agent reports failures you cannot find in the raw log.
• Failure counts differ between the agent's summary and the CI dashboard.
• The eval reads a truncated or summarized log instead of the full output.

When not to use

• The failures are real and correctly quoted (the agent did its job; fix the tests).
• The log itself is missing output because the runner crashed mid-write (a logging problem, not a hallucination).

Tool compatibility

• Any eval or coding agent (Claude Code, Cursor, etc.) summarizing pytest, Jest, Vitest, Go test, or JUnit output.
• Needs machine-readable results (JUnit XML, TAP, or JSON reporter) for the automated cross-check.

Variant phrasings

agent reported failures that do not exist

The core symptom: names in the summary, nothing in the log. Verbatim-quote requirements kill it.

eval summary disagrees with the CI dashboard

Two sources, one truth. The parsed machine output is the arbiter; the agent's prose is the suspect.

agent cited a test name that was never collected

Cross-checking against --collect-only catches invented names even when the log is too long to eyeball.

Why it happens

Summarization is pattern completion: faced with a long log and an instruction to "report failures", the agent generates plausible failure-shaped text. Truncated logs make it worse, because the agent fills the unseen middle with invented content instead of saying "I could not see the failures".

Edge cases

• Retried tests appear as failures in raw logs but pass on retry; the machine parse must use final status, or the agent looks like it hallucinated when it actually read stale lines.
• Parameterized test IDs are long and easy to misquote; require the full node ID, not a shortened name.
• Flaky tests fail in the log the agent read but pass on rerun; timestamp the log so "hallucination" is not confused with "already fixed".

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ald8UyDbhgx8m80oyh4-TQ

Resolved from the public thread: https://vectle.com/posts/pst_Ald8UyDbhgx8m80oyh4-TQ

Thread query: eval agent hallucinating test failures not in the log: how to ground it

TL;DR: An eval agent that summarizes test results from a long log will sometimes invent failures, names and all. Ground every claim: require one verbatim quoted log line per reported failure, cross-check test names against the runner's collected list, and add an automated diff between the agent's claim list and the parsed JUnit or TAP output. Claims without quotes get rejected.