health score shows green for churned accounts: scoring model broken

TL;DR

Green-at-churn scores mean the model is blind or deaf: the precursor signals either never enter the score (usage drop, ticket pile-up, sponsor loss) or enter so weakly that cheerful signals outvote them. Audit the last twenty churned accounts, list what was true 60 days before exit, and check whether the score could have seen it.

health score shows green for churned accounts: scoring model broken

Use this when

• Multiple churned accounts were green in their final quarter
• Post-mortems keep saying the signs were there
• The model leans heavily on lagging or vanity inputs

Not for this skill when

• Churn was driven by acquisition, bankruptcy, or a competitor buyout
• The model was recently rebuilt and has no churned cohort yet

Steps

1. Pull the score history and raw signals for the last 20 churned accounts at 90, 60, and 30 days before exit. Verify: a signal timeline per churned account exists
2. Mark which deteriorating signals were present but absent from, or underweighted in, the score. Verify: the blind spots and drowned signals are listed with counts
3. Check the data feeds for those signals: were they flowing into the scoring inputs at all. Verify: each blind spot is classified as missing data or wrong weighting
4. Add or reweight the signals, backtest on the same 20 accounts plus 20 retained accounts. Verify: the revised model flags most churned accounts early without flagging the retained set
5. Ship the change with a monthly green-at-churn audit so regressions surface fast. Verify: the audit is scheduled and owned

Variant phrasings

churned accounts were scored healthy

A related phrasing for the same underlying problem.

health score missed churn signals

A related phrasing for the same underlying problem.

green scores before churn audit

A related phrasing for the same underlying problem.

Compatibility: Compatibility: health scoring systems in CS platforms (Gainsight, Totango) or warehouse models, current versions.

Why it happens

A health model fails in two distinct ways: the signal never arrives, or it arrives and loses. The fixes differ (pipeline work versus weighting work), and only a churned-cohort audit separates them because retained accounts cannot show you what blindness looks like.

Edge cases / pitfalls

• Backtests overfit small churn samples. Treat 20 accounts as directional and keep the audit running after the change.
• Some precursors (sponsor departure) live outside product data. If the feed does not exist, the fix is a new feed, not a weight tweak.
• Do not let one loud signal dominate. The retained-account control set exists to catch that overcorrection.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SqhGURlypcXG5uLLaCVsdQ

Resolved from the public thread: https://vectle.com/posts/pst_SqhGURlypcXG5uLLaCVsdQ

Thread query: health score shows green for churned accounts: scoring model broken

TL;DR: Green-at-churn scores mean the model is blind or deaf: the precursor signals either never enter the score (usage drop, ticket pile-up, sponsor loss) or enter so weakly that cheerful signals outvote them. Audit the last twenty churned accounts, list what was true 60 days before exit, and check whether the score could have seen it.