LLM hallucinating cancellation reasons: agent failed to classify 'other' tickets correctly

TL;DR

An other bucket that swallows a third of cancellations means the classifier cannot tell the labels apart: definitions overlap, examples are missing, or the model gets no context beyond one short text. Audit misclassified samples, rewrite label definitions with boundary examples, require a confidence, and send low-confidence items to humans instead of forcing a guess.

LLM hallucinating cancellation reasons: agent failed to classify 'other' tickets correctly

Use this when

• The other category is the largest or fastest-growing reason
• Spot checks show price, product-gap, and support cases inside other
• Labels were defined as names without descriptions or examples

Not for this skill when

• Customers genuinely give vague reasons and even humans cannot classify them
• The capture flow forces other by design, a form problem upstream

Steps

1. Sample 50 other-classified cancellations and hand-label them with a human. Verify: the true composition of other is measured
2. Rewrite each label's definition with inclusion rules and two boundary examples drawn from real cases. Verify: the taxonomy becomes decidable
3. Re-run the classifier with the new definitions and a required confidence score on the same sample. Verify: accuracy against hand labels improves measurably, or the taxonomy needs another pass
4. Route below-threshold confidence to a human queue and feed their decisions back as examples. Verify: uncertain cases improve the system instead of polluting it
5. Monitor the other share weekly with an alert above an agreed ceiling. Verify: taxonomy rot is caught as it happens

Variant phrasings

LLM dumping cancellations into other

A related phrasing for the same underlying problem.

cancellation reason classification wrong

A related phrasing for the same underlying problem.

taxonomy definitions for classifier

A related phrasing for the same underlying problem.

Compatibility: Compatibility: LLM classification over exit-survey and ticket text (any major model API) with human review tooling, current versions.

Why it happens

Classifiers fail at label boundaries, not at label centres. The other bucket is where boundary failures accumulate invisibly until the data is useless for product decisions. Boundary examples and confidence routing attack exactly the failure's location.

Edge cases / pitfalls

• Hand-labelled samples are the only honest accuracy measure. Model self-reported confidence alone is not evaluation.
• Some texts truly carry two reasons. Allow a primary plus secondary label rather than forcing one.
• Feeding human decisions back as examples must exclude any case the human marked as genuinely unknowable, or other gets trained as a real category.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst1cNguctloZS-XC2gcoI5w