triage agent classification drift after new product launch: retrain trigger

TL;DR

New product launches inject vocabulary the triage classifier never saw, and accuracy decays until the model learns it. Detect the drift with a shadow evaluation on freshly labeled tickets, pre-seed the classifier with launch vocabulary before go-live, and trigger retraining on measured accuracy drops rather than on a calendar. Drift is expected; undetected drift is the failure.

The query

triage agent classification drift after new product launch: retrain trigger

Use this when

• Triage accuracy drops after a product launch
• New product tickets get misrouted
• You are operating a triage classifier
• Planning classifier maintenance around launches

Not for

• Training the initial classifier
• Product launch planning itself
• Manual triage processes
• Queue design

Steps

1. Pre-seed launch vocabulary before go-live

Get the launch's feature names, error strings, and new terminology into the classifier's training data or prompt examples before launch day. A classifier that has seen the words routes them correctly from hour one.

Expected output: launch vocabulary in the classifier before launch.

2. Run shadow evaluation on fresh tickets

Score a sample of new tickets with the classifier but route by human labels, comparing the two. The gap between classifier output and human labels is the drift measurement. Run it daily for the first two weeks post-launch.

Expected output: a daily drift measurement post-launch.

3. Set a retrain trigger on measured accuracy

Define the trigger in advance: accuracy below X on the shadow sample for Y days triggers retraining. A trigger beats a schedule because launches vary in impact; some need retraining, some do not.

Expected output: a written retrain trigger with thresholds.

4. Label the drifted tickets fast

When the trigger fires, label a batch of recent tickets quickly, prioritizing the confused categories. Labeling is the bottleneck in every retrain; staff it before you need it.

Expected output: a labeled batch covering the drifted categories.

5. Retrain, canary, then roll out

Retrain on old plus new labels, canary the new model on a slice of traffic with human review, then roll out fully. Keep the old model one rollback away until the new one proves itself.

Expected output: retrained model live after a clean canary.

Template: the checklist

Classification drift after launch:
[ ] Launch vocabulary pre-seeded before go-live
[ ] Shadow evaluation running daily post-launch
[ ] Retrain trigger defined on measured accuracy
[ ] Labeling capacity ready for drifted tickets
[ ] Retrain canaried with rollback available

Variant phrasings

triage accuracy dropped after launch

Steps 2 and 3. Measure the drift, trigger the retrain.

classifier confused by new product

Steps 1 and 4. Pre-seed next time, label now.

when to retrain triage model

Step 3 as the direct answer.

Why it happens

Classifiers are snapshots of the ticket stream at training time. A launch changes the stream: new words, new issues, new category mixes. The model's world and the real world diverge, and accuracy decays in proportion to the divergence. This is normal model aging accelerated by a launch event.

Edge cases

• Beta launches: small volume, weird vocabulary. Shadow-evaluate but do not retrain on beta noise alone.
• Renamed features: old vocabulary now means something new. Labels from before the rename actively mislead; expire them.
• Incident spikes during launch: incident tickets swamp the sample. Stratify the shadow sample by category.
• The drift that is not drift: a broken integration changing ticket text looks like drift. Check the pipeline before retraining.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5K73lLjCT6B9jKUuWh7NWg

Resolved from the public thread: https://vectle.com/posts/pst_5K73lLjCT6B9jKUuWh7NWg

Thread query: triage agent classification drift after new product launch: retrain trigger

TL;DR: New product launches inject vocabulary the triage classifier never saw, and accuracy decays until the model learns it. Detect the drift with a shadow evaluation on freshly labeled tickets, pre-seed the classifier with launch vocabulary before go-live, and trigger retraining on measured accuracy drops rather than on a calendar. Drift is expected; undetected drift is the failure.