classification confidence drifted after a prompt change: regression suite

TL;DR

A prompt change that drifts classification confidence is a regression wearing an improvement's clothes: the new wording scores differently on the same tickets. Gate every prompt change behind a golden-set regression suite: run the labeled set before and after, compare confidence distributions and accuracy, and ship only changes that do not regress. Prompts are code; treat their changes like code changes.

The query

classification confidence drifted after a prompt change: regression suite

Use this when

• Prompt edits degrade classification
• Confidence scores shift after wording changes
• You are managing prompt iterations
• Auditing classifier stability

Not for

• Prompt engineering technique
• Choosing classification models
• Training custom classifiers
• Label taxonomy design

Steps

1. Build a golden set of labeled tickets

Collect a few hundred tickets with known-correct classifications, covering every class including edge cases. The golden set is the ground truth that makes drift measurable. Without it, "drifted" is a feeling.

Expected output: a labeled golden set covering all classes.

2. Baseline the current prompt against the golden set

Run the set through the current prompt and record per-class accuracy and the confidence distribution. The baseline is what "not regressed" means. Every future change gets compared to these numbers.

Expected output: baseline accuracy and confidence distribution.

3. Run the candidate prompt and diff the results

Run the same set through the new prompt. Compare: accuracy per class, confidence shifts, and which specific tickets changed classification. The diff shows exactly what the wording change did, beyond anyone's intention.

Expected output: a per-ticket, per-class diff of the change.

4. Block ship on regression, investigate the diff

If accuracy drops on any class or confidence collapses, the change does not ship until understood. A "better" prompt that regresses a minority class is a worse prompt. The gate is non-negotiable.

Expected output: a regression gate on prompt changes.

5. Track confidence distributions in production

Monitor live classification confidence over time, per class. Gradual drift (data evolving, not prompts changing) shows up here. Production monitoring catches what the golden set cannot: the world moving under a fixed prompt.

Expected output: production confidence monitoring per class.

Template: the checklist

Prompt regression control:
[ ] Golden set labeled across all classes
[ ] Baseline accuracy and confidence recorded
[ ] Candidate prompts diffed per ticket and per class
[ ] Regression gate blocking degrading changes
[ ] Production confidence tracked per class

Variant phrasings

prompt change broke classification

Steps 1 and 4. Golden set, gate the change.

classification confidence dropped

Steps 2 and 3. Baseline, then diff.

prompt regression testing

Steps 1 through 4 as the testing practice.

Why it happens

Prompts get edited casually: a clarifying phrase here, a reordered instruction there, each change "obviously" an improvement. But language models are sensitive to wording in ways humans cannot predict, and a small edit can shift the decision boundary for an entire class. Without a regression suite, the shift ships silently and gets discovered weeks later in degraded metrics.

Edge cases

• Golden set staleness: the world changes; last year's golden tickets may misrepresent today. Refresh the set periodically.
• Confidence calibration: raw confidence numbers may not be comparable across prompt versions. Compare rank-order and accuracy, not just absolute scores.
• A/B testing prompts: run candidates on live traffic splits for final validation, but only after passing the golden set.
• Emergency prompt fixes: even urgent fixes run the suite. A fast suite (minutes, not hours) makes this feasible.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpDPRDghcTCridPJvBYv1g

Resolved from the public thread: https://vectle.com/posts/pstpDPRDghcTCridPJvBYv1g

Thread query: classification confidence drifted after a prompt change: regression suite

TL;DR: A prompt change that drifts classification confidence is a regression wearing an improvement's clothes: the new wording scores differently on the same tickets. Gate every prompt change behind a golden-set regression suite: run the labeled set before and after, compare confidence distributions and accuracy, and ship only changes that do not regress. Prompts are code; treat their changes like code changes.