agent hallucinated a competitor migration in churn reasons: classification invented a switch that never happened

TL;DR

An invented reason is worse than a missing one, because a missing reason invites a question and an invented one starts a strategy. When a language model classifies churn reasons, it can complete a plausible story, the customer mentioned evaluating a competitor, so the agent records migrated to that competitor, upgrading a hint into a fact nobody stated. Force classification to cite the exact source text behind every reason, add an explicit unknown outcome the agent is rewarded for using, and audit the reasons already published before product acts on a fictional migration wave.

agent hallucinated a competitor migration in churn reasons: classification invented a switch that never happened

Use this when

• Churn reporting shows competitor migrations that account teams and customers do not recognise
• The source notes behind sampled migration classifications mention evaluation, frustration, or nothing about the competitor at all
• The effect grows where source material is thin, short notes and single calls, where invention fills the most space

Not for this skill when

• Customers really are migrating and the evidence exists in calls or emails the audit has not sampled yet
• The classification taxonomy lacks the true reason, pushing the agent toward the nearest available label, a taxonomy gap rather than invention

Steps

1. Audit a sample against sources. Take twenty accounts classified as competitor migration, pull every source the agent could have used, calls, emails, tickets, notes, and have a human mark each classification as stated, implied, or absent in the sources. The absent rate is your hallucination measure and the number leadership needs before trusting any reason in the current report. Verify: the sample audit produces a stated, implied, absent breakdown for the migration reason
2. Read the classification prompt and rules the agent follows. Look for instructions that force a choice among reasons with no unknown option, examples that model confident labelling on thin evidence, and any step where the agent summarises first and classifies its own summary, compounding invention, because a model asked to always answer will always answer, with fiction when facts run out. Verify: the forcing elements in the instructions are identified
3. Rebuild classification on cited evidence. Require the agent to output the reason together with the exact quoted passage supporting it and its source, reject classifications whose quote does not support the label under a simple checker or a human review queue, and make no evidence a first class outcome labelled unknown, reported openly as its own share rather than redistributed among the plausible reasons. Verify: a test set of thin and clear cases classifies with quotes, and thin cases land on unknown instead of inventing reasons
4. Fix the taxonomy gaps the audit exposed. Where the true reasons in sources have no label, such as internal champion loss recorded as competitor activity, add or split labels with the churn reporting owner, because agents squeeze reality into the nearest available box when the right box does not exist, and the squeeze looks exactly like hallucination. Verify: the taxonomy is updated with the missing reasons and the test set reclassifies correctly
5. Correct the published record. Reclassify the affected window with the grounded process, restate the churn reason report with the unknown share visible, and specifically retract the migration claim to whoever received it, product, leadership, or customers in QBRs, with the corrected figures, because a fictional competitor wave can redirect a roadmap and the retraction must travel as far as the claim did. Verify: the restated report is published and the retraction reaches the original audiences
6. Institutionalise reason audits. Sample classifications monthly against sources with a human reviewer, track the absent rate per reason as the agents quality metric, and require evidence quotes to be stored with every classification permanently, so any future reason can be challenged and checked in minutes rather than by a new investigation. Verify: the monthly audit is scheduled and classifications store their evidence quotes

Why it happens

Language models are trained to produce plausible continuations, and classification prompts that demand one label from a list turn that talent into a liability on thin evidence. A customers passing mention of a competitor is a strong narrative hook, and the model completes it into the migration the business fears most, because that is the story shape it has seen most often. Without an evidence requirement and an allowed unknown, the pipeline manufactures certainty at scale, and churn reasons, which leadership treats as customer voice, become partly fiction.

Compatibility: Churn reason classification by language model agents, source evidence in calls, emails, and tickets, and classification quality auditing, in current agent framework versions.

Variants and edge cases

agent hallucinated competitor migration churn reasons

['The direct phrasing. Evidence quotes or it did not happen, that rule converts an unmeasurable trust problem into a checkable artefact.']

LLM invented churn reason classification no evidence

['The grounding variant. An explicit unknown category is not a failure state, it is the honest output that keeps the known categories believable.']

churn reason report wrong competitor switches

['The strategy variant. Retract as far as the claim travelled, a quiet correction in the data does not undo a loud conclusion in a planning meeting.']

Edge cases to work through before you close the incident:

• Implied is the contested middle. Decide with the reporting owner how implication is labelled, and keep it visibly separate from stated reasons rather than merging them for a cleaner chart.
• One angry support interaction can dominate thin sources. Weight classification by source coverage and show low evidence accounts as such in the report.
• Competitor names are sticky. Once a fictional migration enters a QBR or a roadmap document it gets cited onward, so the correction step must search for the claim in documents, not only in the data.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_y-54OY-hDcH-jfTqq4lplw