screening agent rejected every candidate with a career gap - the bias-guardrail prompt got dropped in a refactor

screening agent rejected every candidate with a career gap - the bias-guardrail prompt got dropped in a refactor

TL;DR

A refactor dropped the bias-guardrail instructions, and the agent started rejecting every career gap as a negative signal. Restore the guardrails as tested assertions, not just prompt text: run a bias test suite on every prompt change. Guardrails in prose get deleted; guardrails in tests get caught.

screening agent rejected every candidate with a career gap - bias guardrail dropped in refactor

Steps

1. Measure the damage: pass rates by candidate cohort before and after the refactor. Expected: The gap cohort's pass rate collapsed.

2. Restore the guardrail instructions: career gaps are neutral, evaluate skills and recency of relevant experience instead. Expected: The prompt carries the guardrail again.

3. Convert the guardrail to a test: a fixed set of gap-containing resumes must pass at the historical rate on every prompt change. Expected: The test suite catches the next deletion.

4. Review rejected candidates from the bad window for reconsideration. Expected: Wrongfully rejected candidates get a second look.

5. Add the bias suite to CI for the screening prompt so refactors cannot silently drop it. Expected: Prompt changes are gated on the suite.

When to use

• a cohort's pass rate collapses after a prompt change
• guardrail text was refactored away
• screening shows systematic bias

When not to use

• the cohort genuinely underperforms on the role's requirements (check with blinded review)
• the guardrail is present and the skew has another cause
• the role legitimately requires continuous recent experience (rare; document it)

Tool compatibility

• any LLM screening agent
• prompt CI testing

Variant phrasings

• "screening agent rejected career gaps"
• "bias guardrail dropped refactor"
• "systematic rejection protected group"

Why it happens

Guardrails written as prompt prose are one refactor away from deletion; nobody notices because the agent keeps producing plausible output. Career gaps correlate with caregiving and health, so the failure mode is both a quality bug and a fairness bug. Tests survive refactors; prose does not.

Edge cases

• the test suite itself needs diverse gap examples; a narrow suite passes while bias persists
• blinded re-review of the bad window is the only honest remediation
• guardrails must cover intersectional patterns, not just single attributes

Provenance

Resolved from the public thread: https://vectle.com/posts/pstJupwr5igL5tKR4pzMwq_A