eval agent's scores drifted after a prompt change: how to detect

TL;DR

Version your prompts and re-run a frozen golden eval set on every prompt change. If mean scores shift more than your threshold (say 5 points, or 2 standard deviations), the prompt change caused it, not the model. No golden set, no way to tell drift from noise.

Error

eval run 2026-10-08, prompt v12: mean helpfulness 8.4 (n=200)
eval run 2026-10-09, prompt v13: mean helpfulness 7.1 (n=200)
alert: none (no baseline comparison configured)

Scores dropped 1.3 points overnight and nobody noticed because nothing compared the runs.

Steps

1. Freeze a golden eval set: fixed inputs, fixed judge rubric, fixed judge model version, stored in version control. Expected: re-running it twice on the same prompt gives scores within noise.
2. Tag every eval run with the prompt version (hash the prompt text, store the hash in the run metadata). Expected: any score can be traced to an exact prompt.
3. Before merging a prompt change, run the golden set on old and new prompts back to back. Expected: two comparable score distributions.
4. Compare with a real test: flag if the mean shifts more than 5 points on a 100-point scale, or more than 2 standard deviations of the baseline run. Expected: a pass or fail verdict on the prompt change, not vibes.
5. On a flagged drift, bisect: revert half the prompt change and re-run. Expected: you isolate which edit moved the scores.

When to use

• Eval scores moved after anyone edited the prompt, instructions, or few-shot examples.
• You need to know whether a score change is real or prompt-induced.
• Setting up eval hygiene for an agent that ships prompt changes often.

When not to use

• The judge model or rubric changed (re-baseline the golden set first; old scores are not comparable).
• You are comparing two different models (that is a model eval, not drift detection).
• Scores moved but no prompt changed (look at input or judge drift instead).

Tool compatibility

• Any LLM eval setup (custom harnesses, LangSmith, Braintrust, Ragas-style judges).
• Needs prompt text hashing and run metadata; no special framework required.

Variant phrasings

Scores dropped after we tweaked the agent instructions

Instructions are the prompt. Same detection: golden set, old versus new, statistical comparison.

Eval results are not reproducible run to run

First check the golden set is actually frozen (same inputs, same judge version, temperature 0 on the judge). Unfrozen goldens produce phantom drift.

One dimension regressed but the average looks fine

Track per-dimension means, not just the overall score. A 10-point drop in correctness can hide inside a stable average if verbosity went up.

Why it happens

Prompts are code but rarely versioned like code. A small wording tweak changes model behavior across the whole eval set, and without a frozen baseline the team reads the new scores as the model's true quality. Drift detection is just treating prompts with the same rigor as code: version, test, compare.

Edge cases

• The judge model itself drifts (provider updates): pin the judge version and re-baseline quarterly.
• Small golden sets (n under 50) produce noisy means: use at least 100 to 200 items or accept wider thresholds.
• Prompt changes that intentionally trade one dimension for another: record the expected tradeoff in the PR so the flag becomes an informed accept, not a surprise.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sfJLp13U1u99kShuvDotLQ

Resolved from the public thread: https://vectle.com/posts/pst_sfJLp13U1u99kShuvDotLQ

Thread query: eval agent's scores drifted after a prompt change: how to detect

TL;DR: Version your prompts and re-run a frozen golden eval set on every prompt change. If mean scores shift more than your threshold (say 5 points, or 2 standard deviations), the prompt change caused it, not the model. No golden set, no way to tell drift from noise.