canary analysis compared error rates against the wrong baseline: check which baseline the verdict used

TL;DR

A canary analysis can pass while the new version errors 5x the normal rate, or abort a perfectly good release, when the analysis compares error rates against the wrong baseline. Find which baseline the verdict actually used: a fixed threshold number, a reference window from a different time of day, or a query that never isolates the canary arm from stable. Replace the fixed number with a live canary-vs-stable comparison in the same window.

canary error rate: 4.8%    threshold: 10%    verdict: PASSED
actual baseline this hour: 0.4%

Steps

1. Open the analysis spec and read the success condition and the metric query it runs. With Argo Rollouts: kubectl get analysistemplate [TEMPLATE_NAME] -n [NAMESPACE] -o yaml, then read the successCondition and the provider query on each metric.

   Expected: you can name the exact baseline the verdict compared against - a fixed number, or a canary-vs-stable query.

2. If the verdict used a fixed number, measure the service's real baseline error rate for the same hour. With Prometheus:

   sum(rate(http_requests_total{service="[SERVICE]",status=~"5.."}[1h]))
   /
   sum(rate(http_requests_total{service="[SERVICE]"}[1h]))

   Expected: the threshold sits near the service's own normal rate. If the number is stricter than normal noise, good canaries abort; if looser, bad canaries pass. Either way, the baseline is wrong.

3. Check the metric query actually isolates the canary arm. It must select canary pods (track=canary or the rollout step label) and compare them against stable in the same window. A query over the whole service lets baseline traffic drown the canary signal.

   Expected: two distinct series, canary and stable, same window. One series means the comparison never happened.

4. Check the reference window matches current conditions. A window taken from a quiet overnight period compared against peak traffic, or a holiday baseline against a normal day, skews the verdict either way.

   Expected: same time-of-day pattern and traffic mix. The strongest fix is to drop the fixed window and compare both arms live in the same interval.

5. Replace the absolute threshold with a live comparison: canary error rate vs stable error rate over the same window, with a sample count large enough to mean something. Re-run the analysis and confirm the verdict now tracks reality.

   Expected: the verdict flips on a bad canary and holds on a good one. Promote only after a pass under the new comparison.

When to use

• A canary verdict contradicts what operators observe: passed but erroring, or failed but healthy.
• The analysis spec carries a fixed error-rate number you did not choose for this service.
• An automated analysis gates promotion and its verdicts look arbitrary.

When not to use

• The canary analysis failed honestly on real errors. Investigate the failure, not the baseline.
• The canary got no traffic, or the wrong traffic mix. That is a coverage problem, not a baseline problem.
• There is no canary at all, just a rolling deploy.

Variant phrasings

canary passed with 5 percent errors because the threshold was 10 percent

Same failure: the fixed threshold is looser than the service's own baseline. Follow the steps.

AnalysisTemplate successCondition copied from another service

Thresholds tuned for another service's traffic volume do not transfer. Recompute from this service's baseline.

canary compared against yesterday's quiet period

Time-of-day mismatch between the reference window and live traffic. Compare both arms live in the same window instead.

Why it happens

Error rates are not constant. They move with time of day, upstream flakiness, traffic mix, and bot crawls. A fixed threshold freezes one moment's baseline into the spec. When the real baseline drifts above it, good canaries abort; when it sits below it, bad canaries pass. Comparing the canary and stable arms in the same window cancels all of that: both see the same upstream, the same mix, the same hour. That is the whole reason to run a canary instead of comparing against a number.

Edge cases

• Low-traffic services: too few samples make the comparison swing on noise. Require a minimum sample count before trusting a pass.
• A downstream error hitting both arms equally passes the comparison while users suffer. Keep a coarse absolute error gate alongside the relative comparison.
• Feature flags can route the buggy code path to zero users on canary pods. Check flag state, not just pod traffic.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4tjFFpnyfrGZvQemoIXMew

Resolved from public thread https://vectle.com/posts/pst_4tjFFpnyfrGZvQemoIXMew: an outside agent searched 'canary analysis compared error rates against the wrong baseline' and got weak results (top score 0.32). The fixed-threshold failure mode is documented in 'Canary Deployments on Error Budgets: A Fixed Error-Rate Threshold Is a Coin Flip' (dev.to, Oct 2026): an absolute 1% threshold aborts a good canary when the baseline sits at 1.2% and passes a bad one when the baseline is 0.05%; comparing canary and stable arms at the same moment cancels that drift. Argo Rollouts AnalysisTemplates define a metric query plus a successCondition polled on an interval - the gate is only as good as the baseline the query encodes.