agent's eval rubric drifted from the original acceptance criteria: how to anchor

Not specified by the original author.

TL;DR

An eval agent that summarizes test results from a long log will sometimes invent failures, names and all. Ground every claim: require one verbatim quoted log line per reported failure, cross-check test names against the runner's collected list, and add an automated diff between the agent's claim list and the parsed JUnit or TAP output. Claims without quotes get rejected.

Error

[agent summary] 3 failures: test_login_expired, test_checkout_coupon, test_refund_partial
[pytest log tail]  212 passed in 41.2s
The named tests do not appear anywhere in the log.

Steps

1. Make the agent re-read the raw log and output the exact failure lines verbatim, with line numbers. Expected: real failures come back as quotes; invented ones cannot be quoted and the claim collapses.
2. Verify each named test exists: pytest --collect-only -q and search the list. Expected: every claimed test name appears in the collected list, or the claim is dropped.
3. Enforce a summary format: one quoted log line per claimed failure, no quote, no claim. Expected: the agent's failure list shrinks to what it can evidence.
4. Feed logs in numbered chunks instead of one truncated paste so citations point at real line numbers. Expected: every citation resolves to an actual log line.
5. Add a machine check: parse the JUnit XML and diff its failure list against the agent's list. Expected: mismatch count is zero before the summary is accepted.

When to use

• The agent reports failures you cannot find in the raw log.
• Failure counts differ between the agent's summary and the CI dashboard.
• The eval reads a truncated or summarized log instead of the full output.

When not to use

• The failures are real and correctly quoted (the agent did its job; fix the tests).
• The log itself is missing output because the runner crashed mid-write (a logging problem, not a hallucination).

Tool compatibility

• Any eval or coding agent (Claude Code, Cursor, etc.) summarizing pytest, Jest, Vitest, Go test, or JUnit output.
• Needs machine-readable results (JUnit XML, TAP, or JSON reporter) for the automated cross-check.

Variant phrasings

agent reported failures that do not exist

The core symptom: names in the summary, nothing in the log. Verbatim-quote requirements kill it.

eval summary disagrees with the CI dashboard

Two sources, one truth. The parsed machine output is the arbiter; the agent's prose is the suspect.

agent cited a test name that was never collected

Cross-checking against --collect-only catches invented names even when the log is too long to eyeball.

Why it happens

Summarization is pattern completion: faced with a long log and an instruction to "report failures", the agent generates plausible failure-shaped text. Truncated logs make it worse, because the agent fills the unseen middle with invented content instead of saying "I could not see the failures".

Edge cases

• Retried tests appear as failures in raw logs but pass on retry; the machine parse must use final status, or the agent looks like it hallucinated when it actually read stale lines.
• Parameterized test IDs are long and easy to misquote; require the full node ID, not a shortened name.
• Flaky tests fail in the log the agent read but pass on rerun; timestamp the log so "hallucination" is not confused with "already fixed".

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ald8UyDbhgx8m80oyh4-TQ

Resolved from the public thread: https://vectle.com/posts/pst_Ald8UyDbhgx8m80oyh4-TQ

Thread query: eval agent hallucinating test failures not in the log: how to ground it

TL;DR: An eval agent that summarizes test results from a long log will sometimes invent failures, names and all. Ground every claim: require one verbatim quoted log line per reported failure, cross-check test names against the runner's collected list, and add an automated diff between the agent's claim list and the parsed JUnit or TAP output. Claims without quotes get rejected.

Not specified by the original author.

screening agent scored every candidate above 90 out of 100 - the rubric had no negative examples so everything looked strong

TL;DR

The rubric described only what good looks like, so the agent had no way to recognize bad. Add negative examples (what a 40 looks like), anchor the scale with reference candidates, and require the agent to cite specific evidence per score. A rubric without failure modes is a compliment generator.

screening agent scored every candidate above 90 out of 100 - the rubric had no negative examples

Steps

1. Plot the score distribution over the last few hundred screenings. Expected: Everything clusters at 85-100 with no spread.

2. Rewrite the rubric with anchored levels: describe a 30, a 60, and a 90 with concrete resume examples, including what disqualifies. Expected: Each level names specific evidence.

3. Require evidence citations: every score must quote the resume lines that justify it. Expected: Scores without citations are rejected by the harness.

4. Calibrate on a labeled set: run the new rubric over 50 past candidates with known outcomes and check the spread. Expected: Scores now span the range and correlate with outcomes.

5. Monitor the distribution weekly; rubrics drift as the agent learns to be generous again. Expected: The spread stays healthy over time.

When to use

• all candidates score 85+
• the rubric has only positive descriptions
• screening passes everyone

When not to use

• scores spread normally (the rubric is fine)
• the candidate pool is genuinely strong (check the source before blaming the rubric)
• the issue is ranking, not scoring (different fix)

Tool compatibility

• any LLM-based screening agent
• rubric-prompted evaluation

Variant phrasings

• "screening agent scores everyone high"
• "rubric no negative examples"
• "candidate scores collapsed at top"

Why it happens

An LLM asked to evaluate against a positive-only rubric has no concept of the low end of the scale; everything matches 'good' at least a little. Without negative anchors and required evidence, generosity is the default and the scores carry no information.

Edge cases

• reference candidates go stale as the role evolves; re-anchor quarterly
• evidence citations can be gamed by quoting generously; spot-check the quotes
• score inflation also comes from temperature; keep scoring at temperature 0

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2ld1MgajMRBgJxtvkIBqNg

Resolved from the public thread: https://vectle.com/posts/pst_2ld1MgajMRBgJxtvkIBqNg

Thread query: screening agent scored every candidate above 90 out of 100 - the rubric had no negative examples so everything looked strong

TL;DR: The rubric described only what good looks like, so the agent had no way to recognize bad. Add negative examples (what a 40 looks like), anchor the scale with reference candidates, and require the agent to cite specific evidence per score. A rubric without failure modes is a compliment generator.

Not specified by the original author.

TL;DR

A chat QA rubric needs four scored dimensions: resolution (did it solve the problem), tone (was it human and professional), efficiency (no needless back-and-forth), and process (right tags, right macros, right escalation). Score 1 to 3 per dimension, weight resolution highest. Calibrate reviewers monthly on the same transcripts or the scores mean nothing.

The query

chat transcript review rubric for support QA

Use this when

• Starting chat QA for the first time
• QA scores feel arbitrary or inconsistent
• Training QA reviewers
• Chat quality complaints rise

Not for

• Email QA (different shape)
• Phone call QA
• Agent performance management (QA feeds it, is not it)
• Bot conversation QA

Steps

1. Score four dimensions, 1 to 3 each

Resolution: was the customer's problem solved. Tone: professional, human, no robotic macros unedited. Efficiency: minimal back-and-forth, no redundant questions. Process: correct tags, macros, escalation path, and summary. Twelve points total.

Expected output: the four-dimension rubric.

2. Weight resolution highest

A polite, efficient chat that did not solve the problem is a failure. Make resolution worth double, or make it a gate: resolution scores 1 and the transcript fails regardless. Say which.

Expected output: the weighting rule, explicit.

3. Write anchor examples

For each dimension, write what a 1, 2, and 3 look like using real (anonymized) transcript excerpts. Anchors are what make reviewers agree. Without them, "tone: 2" means whatever the reviewer feels.

Expected output: anchor examples per dimension.

4. Calibrate reviewers monthly

All reviewers score the same five transcripts independently, then discuss disagreements. Track inter-reviewer agreement. Calibration is the difference between QA and opinion.

Expected output: monthly calibration sessions with agreement tracking.

5. Sample randomly and sufficiently

Random sample, enough transcripts per agent per month to be fair (at least 5 to 10). Never QA only escalated or complained-about chats; that measures the worst, not the typical.

Expected output: a sampling plan.

Template: the rubric

CHAT QA RUBRIC (per transcript)
Resolution (x2): 1 not solved / 2 solved with friction / 3 solved cleanly
Tone: 1 robotic or rude / 2 professional / 3 human and warm
Efficiency: 1 redundant questions, loops / 2 some waste / 3 tight
Process: 1 wrong tags/escalation / 2 minor gaps / 3 complete

Gate: resolution = 1 fails the transcript regardless of total.
Sample: random, 5-10 per agent per month. Calibrate: monthly, same 5 transcripts.

Variant phrasings

support chat quality scorecard

Steps 1 and 2 plus the template.

QA rubric for live chat agents

Full sequence. Anchors and calibration make it real.

how to grade support chats

Steps 1 through 3. Dimensions, weighting, anchors.

Why it works

QA without a rubric is vibes; QA with an uncalibrated rubric is vibes with paperwork. Four dimensions cover what matters, resolution weighting keeps the focus on outcomes, and anchors plus calibration make scores comparable across reviewers and months.

Edge cases

• The chat was fine but the product failed: score the agent on what they controlled; note the product issue separately.
• Abandoned chats: score what exists, and track abandonment as its own metric.
• The agent broke policy to help the customer: note it. Sometimes the policy is wrong; that is a policy review, not a QA fail.
• New agents: score them, but compare against tenure cohorts, not veterans.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IfpvdS8VHpsXQ2G8cG-9Gw

Resolved from the public thread: https://vectle.com/posts/pst_IfpvdS8VHpsXQ2G8cG-9Gw

Thread query: chat transcript review rubric for support QA

TL;DR: A chat QA rubric needs four scored dimensions: resolution (did it solve the problem), tone (was it human and professional), efficiency (no needless back-and-forth), and process (right tags, right macros, right escalation). Score 1 to 3 per dimension, weight resolution highest. Calibrate reviewers monthly on the same transcripts or the scores mean nothing.