canary analysis "insufficient data" errors: how to fix metric gaps

TL;DR

"Insufficient data" means the analysis window lacks the metrics to compare: the canary got no traffic, the metrics have gaps, or the baseline period has no data for a new service. Fix it by ensuring the canary actually receives traffic (synthetic if needed), aligning metric windows with data availability, and giving new services a baseline period before canarying. An analysis with no data is honest; forcing it is not.

The query

canary analysis "insufficient data" errors: how to fix metric gaps

Use this when

• Canary analysis fails with insufficient data
• New services cannot establish baselines
• Metric gaps invalidate comparisons
• After changing metric collection

Not for when

• Canary promotion/rollback strategy
• Metric query correctness (different issue)
• Load testing

Steps

Step 1: Verify the canary received traffic

Check request counts for the canary during the analysis window. Zero traffic means zero data; the fix is traffic (real or synthetic), not analysis tuning. New endpoints with no callers cannot be canaried. Expected output: canary traffic confirmed present, or the traffic gap found.

Step 2: Check metric continuity in the window

Look for gaps in the metrics during baseline and canary windows: collection outages, new metric names after a deploy, or retention boundaries. Gaps invalidate the statistical comparison honestly. Expected output: the metric gaps located and explained.

Step 3: Align windows with data availability

Shorten or shift the analysis windows to periods with complete data. A 30-minute analysis over a window with 20 minutes of data is the common shape of this error. Expected output: windows covering only well-measured periods.

Step 4: Establish baselines for new services

New services need a soak period with traffic before canary analysis means anything. Run the baseline version under load first, collect the metrics, then start canarying changes against it. Expected output: a real baseline existing before the first canary.

Step 5: Add synthetic traffic where real traffic is thin

For low-traffic services, synthetic checks generate the steady baseline canary analysis needs. Real user traffic is ideal; consistent synthetic traffic beats sparse real traffic for statistical validity. Expected output: sufficient data density for reliable analysis.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6F7LfTFIRSfv9KyqD4YpPg