## TL;DR

A canary analysis can pass while the new version errors 5x the normal rate, or abort a perfectly good release, when the analysis compares error rates against the wrong baseline. Find which baseline the verdict actually used: a fixed threshold number, a reference window from a different time of day, or a query that never isolates the canary arm from stable. Replace the fixed number with a live canary-vs-stable comparison in the same window.

```text
canary error rate: 4.8%    threshold: 10%    verdict: PASSED
actual baseline this hour: 0.4%
```

## Steps

1. Open the analysis spec and read the success condition and the metric query it runs. With Argo Rollouts: `kubectl get analysistemplate [TEMPLATE_NAME] -n [NAMESPACE] -o yaml`, then read the `successCondition` and the provider query on each metric.

   Expected: you can name the exact baseline the verdict compared against - a fixed number, or a canary-vs-stable query.

2. If the verdict used a fixed number, measure the service's real baseline error rate for the same hour. With Prometheus:

   ```text
   sum(rate(http_requests_total{service="[SERVICE]",status=~"5.."}[1h]))
   /
   sum(rate(http_requests_total{service="[SERVICE]"}[1h]))
   ```

   Expected: the threshold sits near the service's own normal rate. If the number is stricter than normal noise, good canaries abort; if looser, bad canaries pass. Either way, the baseline is wrong.

3. Check the metric query actually isolates the canary arm. It must select canary pods (track=canary or the rollout step label) and compare them against stable in the same window. A query over the whole service lets baseline traffic drown the canary signal.

   Expected: two distinct series, canary and stable, same window. One series means the comparison never happened.

4. Check the reference window matches current conditions. A window taken from a quiet overnight period compared against peak traffic, or a holiday baseline against a normal day, skews the verdict either way.

   Expected: same time-of-day pattern and traffic mix. The strongest fix is to drop the fixed window and compare both arms live in the same interval.

5. Replace the absolute threshold with a live comparison: canary error rate vs stable error rate over the same window, with a sample count large enough to mean something. Re-run the analysis and confirm the verdict now tracks reality.

   Expected: the verdict flips on a bad canary and holds on a good one. Promote only after a pass under the new comparison.

## When to use

- A canary verdict contradicts what operators observe: passed but erroring, or failed but healthy.
- The analysis spec carries a fixed error-rate number you did not choose for this service.
- An automated analysis gates promotion and its verdicts look arbitrary.

## When not to use

- The canary analysis failed honestly on real errors. Investigate the failure, not the baseline.
- The canary got no traffic, or the wrong traffic mix. That is a coverage problem, not a baseline problem.
- There is no canary at all, just a rolling deploy.

## Variant phrasings

### canary passed with 5 percent errors because the threshold was 10 percent

Same failure: the fixed threshold is looser than the service's own baseline. Follow the steps.

### AnalysisTemplate successCondition copied from another service

Thresholds tuned for another service's traffic volume do not transfer. Recompute from this service's baseline.

### canary compared against yesterday's quiet period

Time-of-day mismatch between the reference window and live traffic. Compare both arms live in the same window instead.

## Why it happens

Error rates are not constant. They move with time of day, upstream flakiness, traffic mix, and bot crawls. A fixed threshold freezes one moment's baseline into the spec. When the real baseline drifts above it, good canaries abort; when it sits below it, bad canaries pass. Comparing the canary and stable arms in the same window cancels all of that: both see the same upstream, the same mix, the same hour. That is the whole reason to run a canary instead of comparing against a number.

## Edge cases

- Low-traffic services: too few samples make the comparison swing on noise. Require a minimum sample count before trusting a pass.
- A downstream error hitting both arms equally passes the comparison while users suffer. Keep a coarse absolute error gate alongside the relative comparison.
- Feature flags can route the buggy code path to zero users on canary pods. Check flag state, not just pod traffic.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4tjFFpnyfrGZvQemoIXMew