screening agent scored every candidate above 90 out of 100 - the rubric had no negative examples so everything looked...

screening agent scored every candidate above 90 out of 100 - the rubric had no negative examples so everything looked strong

TL;DR

The rubric described only what good looks like, so the agent had no way to recognize bad. Add negative examples (what a 40 looks like), anchor the scale with reference candidates, and require the agent to cite specific evidence per score. A rubric without failure modes is a compliment generator.

screening agent scored every candidate above 90 out of 100 - the rubric had no negative examples

Steps

1. Plot the score distribution over the last few hundred screenings. Expected: Everything clusters at 85-100 with no spread.

2. Rewrite the rubric with anchored levels: describe a 30, a 60, and a 90 with concrete resume examples, including what disqualifies. Expected: Each level names specific evidence.

3. Require evidence citations: every score must quote the resume lines that justify it. Expected: Scores without citations are rejected by the harness.

4. Calibrate on a labeled set: run the new rubric over 50 past candidates with known outcomes and check the spread. Expected: Scores now span the range and correlate with outcomes.

5. Monitor the distribution weekly; rubrics drift as the agent learns to be generous again. Expected: The spread stays healthy over time.

When to use

• all candidates score 85+
• the rubric has only positive descriptions
• screening passes everyone

When not to use

• scores spread normally (the rubric is fine)
• the candidate pool is genuinely strong (check the source before blaming the rubric)
• the issue is ranking, not scoring (different fix)

Tool compatibility

• any LLM-based screening agent
• rubric-prompted evaluation

Variant phrasings

• "screening agent scores everyone high"
• "rubric no negative examples"
• "candidate scores collapsed at top"

Why it happens

An LLM asked to evaluate against a positive-only rubric has no concept of the low end of the scale; everything matches 'good' at least a little. Without negative anchors and required evidence, generosity is the default and the scores carry no information.

Edge cases

• reference candidates go stale as the role evolves; re-anchor quarterly
• evidence citations can be gamed by quoting generously; spot-check the quotes
• score inflation also comes from temperature; keep scoring at temperature 0

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2ld1MgajMRBgJxtvkIBqNg