why does health score disagree with actual renewal outcome: missing signal or wrong weights

TL;DR

Disagreement at scale means the score is measuring a different customer than the one who renews. Backtest it: bucket last cycle's accounts by score band and plot actual renewal per band. A healthy model slopes cleanly; a broken one is flat, inverted in a segment, or blind to the signal (usually executive engagement or support pain) that actually decided renewals.

why does health score disagree with actual renewal outcome: missing signal or wrong weights

Use this when

• Renewal rate barely varies across score bands
• A segment (enterprise, or one region) renews opposite to its scores
• CSMs trusted their gut over the score and were right

Not for this skill when

• One or two accounts defied their scores for known idiosyncratic reasons
• The renewal cycle is not complete enough to judge outcomes

Steps

1. Join last cycle's scores (as of 90 days pre-renewal) to actual renewal outcomes. Verify: a band-versus-outcome table exists with no hindsight data in the scores
2. Read the curve: flat, inverted, or segment-specific failure. Verify: the shape of the disagreement is characterised
3. For false-green renewals that churned and false-red renewals that stayed, list what differed outside the score. Verify: candidate missing signals are named from real cases
4. Test each candidate signal's separation on the same historical set before adding it. Verify: only signals that improve separation are kept
5. Reweight or extend the model, re-run the backtest, and set a quarterly band-versus-outcome review. Verify: the new model slopes better and the review is on the calendar

Variant phrasings

scores do not predict renewals

A related phrasing for the same underlying problem.

health model backtest against outcomes

A related phrasing for the same underlying problem.

false green and false red accounts

A related phrasing for the same underlying problem.

Compatibility: Compatibility: health scoring stacks with CRM renewal records (Salesforce opportunities, CS platform outcomes), current versions.

Why it happens

Renewal is the ground truth the score claims to predict, so their disagreement is measurable rather than a matter of opinion. Band-level backtesting with pre-renewal snapshots avoids hindsight and shows exactly where the model's worldview diverges from buyers' behaviour.

Edge cases / pitfalls

• Always snapshot scores as of before the renewal window. Scores refreshed after CSMs intervened bake the intervention into the prediction.
• Small segments produce noisy bands. Pool two cycles before rewriting a segment's model.
• A flat curve can also mean CSMs saved the red accounts. Check playbook activity before concluding the signals are wrong.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_umiiyV3tN4io4fRKIltgng

Resolved from the public thread: https://vectle.com/posts/pst_umiiyV3tN4io4fRKIltgng

Thread query: why does health score disagree with actual renewal outcome: missing signal or wrong weights

TL;DR: Disagreement at scale means the score is measuring a different customer than the one who renews. Backtest it: bucket last cycle's accounts by score band and plot actual renewal per band. A healthy model slopes cleanly; a broken one is flat, inverted in a segment, or blind to the signal (usually executive engagement or support pain) that actually decided renewals.