candidate-matching agent ranked a bootcamp grad above a staff engineer - embedding similarity matched keywords, not...

candidate-matching agent ranked a bootcamp grad above a staff engineer - embedding similarity matched keywords, not seniority

TL;DR

Embeddings match words, not careers: 'Python' in a bootcamp project looks like 'Python' in a decade of production work. Add seniority as an explicit feature (years, scope, titles) and blend it with the similarity score. Keyword overlap is a retrieval signal, not a ranking signal.

candidate-matching agent ranked a bootcamp grad above a staff engineer - embedding similarity matched keywords, not seniority

Steps

1. Inspect the ranking inputs: confirm the score is pure embedding similarity over resume text. Expected: No seniority feature exists in the ranking.

2. Define seniority signals: years of experience, scope words (led, owned, staff, principal), and title progression. Expected: A concrete feature list.

3. Blend: score = alpha similarity + (1 - alpha) seniority_score, tuned on historical hiring data. Expected: Rankings now respect seniority.

4. Validate on known pairs: staff engineers must outrank juniors for senior roles in the test set. Expected: The inversion cases resolve.

5. Log both components per ranking so the next inversion is debuggable. Expected: Each ranking shows its work.

When to use

• junior candidates outrank senior ones
• ranking is pure embedding similarity
• keyword overlap drives the order

When not to use

• the role is genuinely junior (then the ranking may be right)
• seniority signals are already features (tune the blend instead)
• the pool has no senior candidates (ranking cannot invent them)

Tool compatibility

• any embedding-based matcher
• hybrid ranking

Variant phrasings

• "embedding similarity matched keywords not seniority"
• "bootcamp ranked above staff engineer"
• "candidate matching ignores seniority"

Why it happens

Embeddings compress text into similarity, and a bootcamp resume can share most of its vocabulary with a staff resume. Seniority lives in sparse signals (titles, years, scope) that dense similarity washes out. Without an explicit seniority term, the ranker is a keyword counter with extra steps.

Edge cases

• years of experience can be inflated; cross-check with title progression
• over-weighting seniority buries strong juniors for junior roles; blend per role level
• embeddings also miss career gaps and context; they are one signal among several

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0di7RE9pb1pN7Fl3lJJk3Q