NPS drop unexplained: how to find the broken survey pipeline

TL;DR

An NPS cliff with no matching product story is usually the survey, not the sentiment: the send list shrank to unhappy users, invites bounced, or happy responses stopped joining back. Decompose the drop into who was asked, who received, and whose answers were counted before believing the number.

NPS drop unexplained: how to find the broken survey pipeline

Use this when

• NPS dropped sharply in one period
• Response volume or mix changed at the same time
• Usage and support signals do not corroborate a collapse

Not for this skill when

• A known outage or pricing change plausibly explains real detractor growth
• Response volume and mix are stable and verbatim comments match the drop

Steps

1. Compare invites sent, delivered, and responses counted across the last four survey waves. Verify: the stage where volume fell out is located
2. Compare respondent mix (plan, tenure, segment) between the healthy and dropped waves. Verify: any sampling skew toward unhappy segments is quantified
3. Trace 10 raw responses from the survey tool into the NPS table, checking score and segment fields. Verify: join or field-mapping losses are found or ruled out
4. Fix the delivery, sampling, or join fault and recompute the affected wave with corrected data. Verify: the corrected wave is published with the fault annotated
5. Add wave-over-wave guardrails on invite volume and respondent mix that flag anomalies before the score is shared. Verify: future pipeline breaks are flagged before leadership sees a fake drop

Variant phrasings

NPS fell with no product change

A related phrasing for the same underlying problem.

survey pipeline sampling broken

A related phrasing for the same underlying problem.

NPS responses not joining correctly

A related phrasing for the same underlying problem.

Compatibility: Compatibility: survey tools (Delighted, Qualtrics, Typeform) syncing to warehouse or CS platforms, current versions.

Why it happens

NPS is a ratio built on a sample, so it has two failure families: sentiment changed, or the sample changed. Volume and mix forensics separate them cheaply, and verbatim comments act as a lie detector: pipeline breaks produce normal comments under a terrible score.

Edge cases / pitfalls

• Do not silently restate a shared NPS. Executives may have acted on it; the correction needs the same audience as the original.
• Incentivised or re-sent waves genuinely skew mix. Check campaign changes alongside pipeline changes.
• A join fix can resurrect old responses into the wrong wave. Key responses to their invite date, not their sync date.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fkgPZbpyOEwT2Ik73hDFBg

Resolved from the public thread: https://vectle.com/posts/pst_fkgPZbpyOEwT2Ik73hDFBg

Thread query: NPS drop unexplained: how to find the broken survey pipeline

TL;DR: An NPS cliff with no matching product story is usually the survey, not the sentiment: the send list shrank to unhappy users, invites bounced, or happy responses stopped joining back. Decompose the drop into who was asked, who received, and whose answers were counted before believing the number.