## TL;DR
A fallback storm means the confidence threshold is set higher than the model's real accuracy on your traffic, so most inputs fall through. Lower the threshold to match measured accuracy, cap consecutive fallbacks before forcing a human handoff, and treat the storm as a data problem: the fallback logs are the training set the bot needs. A bot that falls back gracefully to a human beats a bot that falls back in circles.

## The query

```text
chatbot flow "intent confidence too low" fallback storm
```

## Use this when

- The bot falls back on a large share of inputs
- Fallback rates spike suddenly
- Users complain the bot never understands them
- You are tuning an existing bot flow

## Not for

- Building a chatbot from scratch
- Training or labeling intents
- Choosing between bot platforms
- Marketing the bot to users

## Steps

### 1. Measure the real confidence distribution

Pull recent traffic and plot the top-intent confidence scores. If the threshold sits above where most scores land, the threshold is wrong, not the users. Set the threshold from the data, typically where precision and recall balance for your top intents.

Expected output: a confidence distribution and a data-backed threshold.

### 2. Cap consecutive fallbacks, then hand off

Allow two fallbacks, then the bot must offer a human or a ticket. A bot that says "I did not understand" three times in a row is manufacturing frustration. The cap is a circuit breaker.

Expected output: a hard fallback cap wired to a handoff.

### 3. Make fallbacks useful, not dead ends

A fallback should offer the top two guessed intents as buttons, plus a human option. "I did not understand" with no path forward is where storms turn into tickets and bad CSAT.

Expected output: fallback responses with intent suggestions and a human path.

### 4. Mine the fallback log weekly

Every fallback is a labeled training example waiting to happen: the user's phrasing plus the intent it should have matched. Review the top fallback phrasings weekly and add them as training phrases or new intents.

Expected output: a weekly review turning fallbacks into training data.

### 5. Watch for sudden storms as an incident signal

A sudden fallback spike usually means something changed: a new product launch with new vocabulary, a broken integration the bot depends on, or a prompt change. Treat spikes as incidents, investigate the change, fix the cause.

Expected output: fallback-rate alerting with spike investigation.

## Template: the tuning runbook

```text
Fallback storm:
[ ] Confidence distribution measured, threshold set from data
[ ] Consecutive-fallback cap with human handoff in place
[ ] Fallbacks offer guessed intents plus a human path
[ ] Weekly fallback-log review feeding training data
[ ] Spike alerting with incident-style investigation
```

## Variant phrasings

### chatbot keeps saying i did not understand

Steps 2 and 3. Cap the loop, make fallbacks useful.

### intent confidence threshold tuning

Step 1. Measure first, then set the threshold.

### chatbot fallback rate too high

Steps 1 and 4. Threshold from data, mine the logs.

## Why it happens

Confidence thresholds get set from vendor defaults or demo data, both of which overstate accuracy on real customer phrasing. Real users typo, ramble, combine questions, and use slang. The gap between demo phrasing and ticket phrasing lands entirely in the fallback bucket, and without a cap it loops.

## Edge cases

- Multilingual traffic: confidence varies by language. Consider per-language thresholds.
- New product launches: vocabulary the bot never saw craters confidence. Pre-seed intents before launch.
- Sarcasm and frustration: low confidence on angry phrasing is fine; route by sentiment, not just confidence.
- Over-tuning: a threshold set too low misroutes confidently. Balance with misroute monitoring, not just fallback rate.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4jtYP7hzXeurGuLOGORhUw