agent's canary analysis passed but users still saw errors: how to check canary coverage
This skill checks canary coverage when canary analysis passes but users still see errors. It covers verifying the canary received real traffic, comparing request mix against stable, and tracing where user errors actually originate. Use it before trusting any green canary verdict during an incident.
TL;DR
A canary analysis passes while users still see errors when the canary does not actually receive the traffic, the errors, or the user segments that are failing. Verify what the canary really covered: traffic share, request mix, error sources, and whether the failing users ever hit it. A green canary that saw none of the bad traffic proves nothing.
canary analysis: PASSED
users: still seeing errorsSteps
- Confirm the canary actually received traffic during the analysis window.
kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL],track=canaryExpected: canary pods existed and were ready for the whole window. Then check the load balancer or ingress weights to confirm the canary got its intended share, not zero because of a weight misconfiguration.
- Compare the request mix hitting the canary against the mix hitting stable.
Expected: same endpoints, same user segments, same regions. A canary that only served health checks or internal traffic while real users hit stable will pass every check and miss every bug.
- Check where the user errors are actually coming from: edge, CDN, a different service, or a client version the canary never served.
kubectl logs deployment/[CANARY_DEPLOY] -n [NAMESPACE] --since=1h --tail=500 | grep -i -E 'error|5[0-9][0-9]'Expected: if the canary logs are clean but users report errors, the errors originate outside the canary's path. Trace one failing user request end to end and see which pods it touched.
- Fix the coverage gap before promoting: correct the traffic weights, include the failing segment, or extend the analysis with the real error query.
Expected: the re-run analysis either fails honestly on the canary (good, it caught it) or passes with the failing traffic actually represented (good, it is really safe). Promote only after the second kind of pass.
When to use
- Canary analysis is green but users report errors on the new version.
- You are reviewing an agent's canary verdict before it promotes.
- Setting up canary analysis for the first time and defining what "pass" means.
When not to use
- The canary analysis itself failed (that is a real signal; investigate the failure).
- There is no canary, just a straight rollout (this is about canary coverage specifically).
- The errors started before the deploy (the canary is innocent; look elsewhere).
Why it happens
Canary analysis compares metrics between canary and stable, but the comparison is only valid if both served comparable traffic. Misconfigured weights, sticky sessions pinning test users to stable, CDN caching shielding the canary, or the bug living in a code path the canary's traffic never exercised all produce a green analysis over an empty or skewed sample. The agent sees "passed" and promotes, because the verdict, not the coverage, is what it was told to optimize.
Edge cases
- Errors from a downstream service affect both canary and stable equally; the comparison passes while users suffer, so canary metrics need absolute error-rate gates too, not just relative ones.
- Short analysis windows miss time-based bugs (cron-triggered, cache-expiry); the window must cover the failing pattern's period.
- Feature flags can route the buggy code path to zero users even on canary pods; check flag state, not just pod traffic.
- A canary that passes on synthetic checks but never saw production traffic is a staging deploy with extra steps; require real-traffic share before trusting the verdict.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_8hSzvobsNg8wINeO-F8RQQ