agent's canary compared preview vs production but preview had stale routes

TL;DR

The canary compared preview and production responses, but preview's routes were stale: it served old code or the wrong worker entirely. The comparison 'passed' or 'failed' on fiction. Verify preview serves the intended version (via a version header) before comparing, and treat route freshness as a precondition of the canary, not an assumption.

The query

agent's canary compared preview vs production but preview had stale routes

Use this when

• Canary comparisons produce confusing or contradictory results.
• Preview serves unexpected content during the canary.
• Route changes happened recently.

Not for

• Canary logic bugs (fix the comparison).
• Genuine behavior differences between preview and production (real findings).
• Preview being down entirely (different problem).

Steps

Step 1: Verify what preview actually serves

Check the version header on the preview URL before trusting any comparison.

Expected result: the served version. If it is not the just-deployed version, the canary measured the wrong thing.

Step 2: Refresh the preview routes

Re-attach or update the preview routes so they point at the current preview deployment.

Expected result: preview serves the intended version. The version header confirms it.

Step 3: Re-run the canary against verified endpoints

Expected result: the comparison is now between the intended versions. Results are meaningful.

Step 4: Make version verification a canary precondition

The canary script checks version headers on both sides first and aborts if either side is not the expected version.

Expected result: stale-route canaries fail fast with a clear message instead of producing confident garbage.

Step 5: Document the preview routing setup

Expected result: how preview routes are created, updated, and torn down is written down. The next agent does not inherit a stale setup silently.

Why it happens

Canary analysis assumes both sides are what they claim to be. Preview environments are ephemeral and their routes are managed separately from code deploys; staleness is the default, freshness the exception. The agent compared labels ('preview' vs 'production') rather than verified versions.

Edge cases

• Stale preview can also serve the PREVIOUS preview version, which may be close enough to pass a loose canary. Tighten the version check to exact match.
• Production can be mid-rollout during the canary too. Verify both sides, not just preview.
• Canary traffic splitting adds a third variable. Keep the canary to two verified endpoints before introducing splits.
• If preview routes are managed by a different team or system, the staleness may recur. Automate the refresh as part of the preview deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_I5yGv6rTWokKiIKVX0hDPg