candidate ranking agent changed its scores between runs - temperature wasn't set to 0 and the rubric drifted

candidate ranking agent changed its scores between runs - temperature wasn't set to 0 and the rubric drifted

TL;DR

The ranking LLM ran at nonzero temperature, so identical resumes scored differently every run, and the rubric drifted between versions. Pin temperature to 0 for scoring, version the rubric, and log both with every score. Non-deterministic scoring is not scoring.

candidate ranking agent changed scores between runs - temperature not 0, rubric drifted

Steps

1. Confirm nondeterminism: score the same resume 10 times and measure the spread. Expected: Scores vary run to run.

2. Pin temperature to 0 (and top_p to 1) for all scoring calls. Expected: The spread collapses.

3. Version the rubric: every score records the rubric version and model version. Expected: Scores are comparable across time.

4. Re-score the affected candidates with the pinned setup and correct the rankings. Expected: Rankings stabilize.

5. Add a determinism check to CI: identical inputs must produce identical scores. Expected: Drift is caught automatically.

When to use

• identical resumes score differently across runs
• temperature is not pinned
• rankings shift without input changes

When not to use

• the rubric intentionally changed (then rescoring is expected; version it)
• the model version changed (different model, different scores; note it)
• scores vary within a tiny band (that is acceptable noise)

Tool compatibility

• any LLM scoring/ranking agent
• temperature and version controls

Variant phrasings

• "ranking scores changed between runs"
• "temperature not zero scoring"
• "rubric drift candidate ranking"

Why it happens

LLM sampling is stochastic unless temperature is 0, and even at 0 some providers vary. Ranking candidates with a nondeterministic scorer means the order depends on sampling luck, and rubric edits between runs silently change the scale. The ranking looked data-driven while being partly random.

Edge cases

• temperature 0 reduces but does not always eliminate variance; verify empirically
• rubric versioning must include the model version; same rubric plus new model can shift scores
• re-scoring historical candidates changes recorded history; annotate, do not rewrite silently

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-orYzoJeyHJB2enFM2BS8A

Resolved from the public thread: https://vectle.com/posts/pst_-orYzoJeyHJB2enFM2BS8A

Thread query: candidate ranking agent changed its scores between runs - temperature wasn't set to 0 and the rubric drifted

TL;DR: The ranking LLM ran at nonzero temperature, so identical resumes scored differently every run, and the rubric drifted between versions. Pin temperature to 0 for scoring, version the rubric, and log both with every score. Non-deterministic scoring is not scoring.