health score weights wrong after product change: recalibration failed

TL;DR

When the product changes, yesterday's healthy behaviour becomes today's anomaly: a consolidated workflow cuts event counts, a new flagship feature deserves weight the old model never gave it. Patching thresholds one complaint at a time never converges. Re-derive the weights from post-change retained and churned behaviour in one pass.

health score weights wrong after product change: recalibration failed

Use this when

• Scores swung after a product launch, consolidation, or packaging change
• CSMs report scores contradicting what they see in accounts
• Old top signals lost volume because the product moved

Not for this skill when

• Only one component is broken, which is a debugging task not a recalibration
• The product change is cosmetic and usage patterns are unchanged

Steps

1. Freeze the behaviour window: define pre and post change periods and exclude the migration weeks. Verify: clean comparison windows are set
2. For post-change data, rank signals by how well they separate retained from churned accounts. Verify: an evidence-ranked signal list replaces the inherited one
3. Draft new weights from that ranking, keeping the total scale and bands familiar to CSMs. Verify: a candidate model exists on the same 0 to 100 scale
4. Backtest the candidate on post-change history and shadow it live for one scoring cycle. Verify: backtest separation improves and shadow scores look sane to account owners
5. Cut over with a changelog, an expected-movement note per segment, and a rollback snapshot. Verify: the new weights ship explained, with the old model restorable

Variant phrasings

health weights stale after product change

A related phrasing for the same underlying problem.

recalibrate scoring after launch

A related phrasing for the same underlying problem.

old usage signals lost meaning

A related phrasing for the same underlying problem.

Compatibility: Compatibility: any health scoring stack (warehouse models or CS platform rules) with retained/churned history, current versions.

Why it happens

Weights encode what the product used to reward. A material product change invalidates that encoding at once, so incremental tweaks chase a moving target. Re-deriving from post-change outcomes is one honest rebuild instead of fifty quiet distortions.

Edge cases / pitfalls

• Post-change history is short at first. Prefer fewer, stronger signals over a rich model fitted on thin data.
• Migration weeks poison both windows: behaviour there is neither old nor new normal. Excluding them is not optional.
• Keep bands and scale stable so CSM playbooks tied to thresholds still make sense after recalibration.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GrC58heNV4J8RS8c7JCAyg

Resolved from the public thread: https://vectle.com/posts/pst_GrC58heNV4J8RS8c7JCAyg

Thread query: health score weights wrong after product change: recalibration failed

TL;DR: When the product changes, yesterday's healthy behaviour becomes today's anomaly: a consolidated workflow cuts event counts, a new flagship feature deserves weight the old model never gave it. Patching thresholds one complaint at a time never converges. Re-derive the weights from post-change retained and churned behaviour in one pass.