dashboard KPI wrong after metric definition change: calculation error troubleshooting

TL;DR

Assume the change only landed in one layer and hunt the others. Metric definitions usually live in the warehouse view, the semantic model, the dashboard calculation and sometimes an extract or cache, and a KPI goes wrong when those copies disagree. Update each copy to the new definition, clear caches and extracts, then reconcile the KPI against a hand written query using the new definition.

dashboard KPI wrong after metric definition change: calculation error troubleshooting

Use this when

• One KPI changed unexpectedly, or failed to change, right after a metric definition update.
• The same KPI shows different values on two dashboards or in the warehouse.
• A hand written query with the new definition does not match the dashboard.

Not for this skill when

• All numbers on the dashboard are stale by the same amount of time. That points at refresh, not definition drift.
• The KPI definition never changed and the value was never right. Treat it as an original build error.

Steps

1. Write down the old and new definitions precisely, including filters, deduplication and the date field used. Verify: both definitions are stated in one paragraph a reviewer can check
2. Run a hand written query for both definitions against the warehouse for a fixed period. Verify: the two results bracket the dashboard value, identifying which definition the dashboard is using
3. Search every layer for the old logic, the warehouse view, the semantic model measure, dashboard calculated fields, extracts and cached tables. Verify: each copy found is listed with its current definition state
4. Update the stale copies to the new definition and refresh extracts and caches that hold computed values. Verify: the refreshed layers report the new definition in their change history
5. Reload the dashboard and compare the KPI with the hand written new definition result for the same filters. Verify: the dashboard KPI matches the reference query exactly
6. Record the definition change in the team metric catalog with an effective date so future gaps are explainable. Verify: the catalog entry names the KPI, the change and the date it took effect

Why it happens

Metrics drift because definitions are copied, not referenced, in most BI stacks. Every copy is a future inconsistency, and definition changes are the moment the copies diverge. Centralizing the definition in one modelled layer, and treating dashboard level calculations as exceptions to retire, is the structural fix. Until then, a layer by layer sweep after each change is the reliable routine.

A definition change, such as counting only paid orders or excluding test accounts, has to propagate through every layer that computes the number. Dashboards often read a cached extract or a duplicated calculated field, so the displayed KPI keeps the old logic while reports built directly on the updated model show the new one. The disagreement pattern, one number right in one place and wrong in another, is the diagnostic key.

Field notes for dashboard estates generally. The most valuable incident habit is to separate the layers before changing anything, source data, dataset or model, dashboard definition, and delivery or viewing layer, and to prove which layer is wrong with one test per layer. Most dashboard failures are reported at the viewing layer and fixed at the wrong layer at least once before the real cause is found. A one line layer verdict in the incident note, with the test that proved it, saves the next engineer the same detour when the symptom recurs.

A second general habit is to keep a known good reference for every important dashboard, one number per dashboard that an owner can check against the source system in under a minute. Reconciliation beats error hunting for the silent failures, wrong definitions, shifted dates, stale caches, because those produce no error at all. Dashboards with a reference check get their silent failures found by the team, while dashboards without one get them found by the audience, which is the difference between maintenance and incident response.

Variants and edge cases

KPI wrong after metric definition change dashboard vs warehouse

The comparison phrasing when two surfaces disagree.

dashboard still using old metric logic extract cache

The stale-copy variant naming extract and cache layers.

metric redefined KPI did not change troubleshooting

The no-change variant where the update never reached the dashboard.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SA0EPUhXgbWrBFg2i5jiWA