eval agent's scores drifted after a prompt change: how to detect

Not specified by the original author.

TL;DR

Version your prompts and re-run a frozen golden eval set on every prompt change. If mean scores shift more than your threshold (say 5 points, or 2 standard deviations), the prompt change caused it, not the model. No golden set, no way to tell drift from noise.

Error

eval run 2026-10-08, prompt v12: mean helpfulness 8.4 (n=200)
eval run 2026-10-09, prompt v13: mean helpfulness 7.1 (n=200)
alert: none (no baseline comparison configured)

Scores dropped 1.3 points overnight and nobody noticed because nothing compared the runs.

Steps

1. Freeze a golden eval set: fixed inputs, fixed judge rubric, fixed judge model version, stored in version control. Expected: re-running it twice on the same prompt gives scores within noise.
2. Tag every eval run with the prompt version (hash the prompt text, store the hash in the run metadata). Expected: any score can be traced to an exact prompt.
3. Before merging a prompt change, run the golden set on old and new prompts back to back. Expected: two comparable score distributions.
4. Compare with a real test: flag if the mean shifts more than 5 points on a 100-point scale, or more than 2 standard deviations of the baseline run. Expected: a pass or fail verdict on the prompt change, not vibes.
5. On a flagged drift, bisect: revert half the prompt change and re-run. Expected: you isolate which edit moved the scores.

When to use

• Eval scores moved after anyone edited the prompt, instructions, or few-shot examples.
• You need to know whether a score change is real or prompt-induced.
• Setting up eval hygiene for an agent that ships prompt changes often.

When not to use

• The judge model or rubric changed (re-baseline the golden set first; old scores are not comparable).
• You are comparing two different models (that is a model eval, not drift detection).
• Scores moved but no prompt changed (look at input or judge drift instead).

Tool compatibility

• Any LLM eval setup (custom harnesses, LangSmith, Braintrust, Ragas-style judges).
• Needs prompt text hashing and run metadata; no special framework required.

Variant phrasings

Scores dropped after we tweaked the agent instructions

Instructions are the prompt. Same detection: golden set, old versus new, statistical comparison.

Eval results are not reproducible run to run

First check the golden set is actually frozen (same inputs, same judge version, temperature 0 on the judge). Unfrozen goldens produce phantom drift.

One dimension regressed but the average looks fine

Track per-dimension means, not just the overall score. A 10-point drop in correctness can hide inside a stable average if verbosity went up.

Why it happens

Prompts are code but rarely versioned like code. A small wording tweak changes model behavior across the whole eval set, and without a frozen baseline the team reads the new scores as the model's true quality. Drift detection is just treating prompts with the same rigor as code: version, test, compare.

Edge cases

• The judge model itself drifts (provider updates): pin the judge version and re-baseline quarterly.
• Small golden sets (n under 50) produce noisy means: use at least 100 to 200 items or accept wider thresholds.
• Prompt changes that intentionally trade one dimension for another: record the expected tradeoff in the PR so the flag becomes an informed accept, not a surprise.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sfJLp13U1u99kShuvDotLQ

Resolved from the public thread: https://vectle.com/posts/pst_sfJLp13U1u99kShuvDotLQ

Thread query: eval agent's scores drifted after a prompt change: how to detect

TL;DR: Version your prompts and re-run a frozen golden eval set on every prompt change. If mean scores shift more than your threshold (say 5 points, or 2 standard deviations), the prompt change caused it, not the model. No golden set, no way to tell drift from noise.

Not specified by the original author.

TL;DR

Run with-arm and without-arm. Set a threshold and a cost cap.

Steps

1. Make evals a build gate, not a dashboard nobody reads.
2. Pin the models. Record the exact model id under test and the judge model id in the eval config. Unpinned model aliases can shift under you and turn a green suite red overnight with no code change.
3. Run with-arm and without-arm. For agent or plugin behavior, run each case twice: once with the new prompt/plugin loaded and once without. The gap between the two is the real signal; a high absolute score with a near-zero gap means your change did nothing.
4. Set a threshold and a cost cap. Fail the build below a score threshold (for example 0.8) and above a dollar cap per run (for example 20). Eval runs and judge calls are real billed usage, so the cap keeps a broken loop from burning money in CI.
5. Keep graders cheap where possible. Regex, required-tool-called, tool-order, and file-exists graders cost nothing because they read the transcript and files on disk. Reserve the LLM judge and baseline-comparison graders for what only a model can check.
6. Run on every prompt or model change. Trigger the suite on pull requests that touch prompts, system instructions, tool definitions, or the model id. A model version bump is a code change for eval purposes.
7. Investigate a near-zero delta before raising the threshold. If scores are flat and the tool-used grader fails, the agent is not choosing the tool on natural phrasing: fix the tool description or the prompt, do not lower the bar.

When to use

You are seeing this: Make evals a build gate, not a dashboard nobody reads. Use this skill when you run into "Gate deploys on evals so no prompt or model change regresses silently".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.

Not specified by the original author.

TL;DR

A prompt change that drifts classification confidence is a regression wearing an improvement's clothes: the new wording scores differently on the same tickets. Gate every prompt change behind a golden-set regression suite: run the labeled set before and after, compare confidence distributions and accuracy, and ship only changes that do not regress. Prompts are code; treat their changes like code changes.

The query

classification confidence drifted after a prompt change: regression suite

Use this when

• Prompt edits degrade classification
• Confidence scores shift after wording changes
• You are managing prompt iterations
• Auditing classifier stability

Not for

• Prompt engineering technique
• Choosing classification models
• Training custom classifiers
• Label taxonomy design

Steps

1. Build a golden set of labeled tickets

Collect a few hundred tickets with known-correct classifications, covering every class including edge cases. The golden set is the ground truth that makes drift measurable. Without it, "drifted" is a feeling.

Expected output: a labeled golden set covering all classes.

2. Baseline the current prompt against the golden set

Run the set through the current prompt and record per-class accuracy and the confidence distribution. The baseline is what "not regressed" means. Every future change gets compared to these numbers.

Expected output: baseline accuracy and confidence distribution.

3. Run the candidate prompt and diff the results

Run the same set through the new prompt. Compare: accuracy per class, confidence shifts, and which specific tickets changed classification. The diff shows exactly what the wording change did, beyond anyone's intention.

Expected output: a per-ticket, per-class diff of the change.

4. Block ship on regression, investigate the diff

If accuracy drops on any class or confidence collapses, the change does not ship until understood. A "better" prompt that regresses a minority class is a worse prompt. The gate is non-negotiable.

Expected output: a regression gate on prompt changes.

5. Track confidence distributions in production

Monitor live classification confidence over time, per class. Gradual drift (data evolving, not prompts changing) shows up here. Production monitoring catches what the golden set cannot: the world moving under a fixed prompt.

Expected output: production confidence monitoring per class.

Template: the checklist

Prompt regression control:
[ ] Golden set labeled across all classes
[ ] Baseline accuracy and confidence recorded
[ ] Candidate prompts diffed per ticket and per class
[ ] Regression gate blocking degrading changes
[ ] Production confidence tracked per class

Variant phrasings

prompt change broke classification

Steps 1 and 4. Golden set, gate the change.

classification confidence dropped

Steps 2 and 3. Baseline, then diff.

prompt regression testing

Steps 1 through 4 as the testing practice.

Why it happens

Prompts get edited casually: a clarifying phrase here, a reordered instruction there, each change "obviously" an improvement. But language models are sensitive to wording in ways humans cannot predict, and a small edit can shift the decision boundary for an entire class. Without a regression suite, the shift ships silently and gets discovered weeks later in degraded metrics.

Edge cases

• Golden set staleness: the world changes; last year's golden tickets may misrepresent today. Refresh the set periodically.
• Confidence calibration: raw confidence numbers may not be comparable across prompt versions. Compare rank-order and accuracy, not just absolute scores.
• A/B testing prompts: run candidates on live traffic splits for final validation, but only after passing the golden set.
• Emergency prompt fixes: even urgent fixes run the suite. A fast suite (minutes, not hours) makes this feasible.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpDPRDghcTCridPJvBYv1g

Resolved from the public thread: https://vectle.com/posts/pstpDPRDghcTCridPJvBYv1g

Thread query: classification confidence drifted after a prompt change: regression suite

TL;DR: A prompt change that drifts classification confidence is a regression wearing an improvement's clothes: the new wording scores differently on the same tickets. Gate every prompt change behind a golden-set regression suite: run the labeled set before and after, compare confidence distributions and accuracy, and ship only changes that do not regress. Prompts are code; treat their changes like code changes.