agent preview deploy overwrote production routes: route collision

Not specified by the original author.

agent preview deploy overwrote production routes: route collision

TL;DR

The preview deploy's routes overlapped production's, so preview stole prod traffic (or vice versa). Scope preview routes to preview hostnames only, never overlapping prod patterns. Previews must be incapable of matching prod traffic.

agent preview deploy overwrote production routes: route collision

Steps

1. Identify the overlapping route patterns. Expected: The collision is mapped.

2. Restrict preview routes to preview-only hostnames/paths. Expected: The overlap is gone.

3. Verify prod traffic routes correctly and preview is isolated. Expected: Both serve correctly.

4. Add a route-collision check to the preview pipeline. Expected: Collisions are caught pre-deploy.

5. Document the route namespacing convention. Expected: The rule is explicit.

When to use

• preview deploys affect production traffic
• routes overlap
• the agent manages previews

When not to use

• the overlap is intentional (canary; then it is not a preview)
• routes are already isolated (check DNS instead)
• the traffic shift is from something else (verify)

Tool compatibility

• wrangler
• preview deployments

Variant phrasings

• "preview deploy route collision"
• "preview overwrote production routes"
• "preview stole prod traffic"

Why it happens

Preview deployments share the routing table namespace, and a preview with a broad pattern matches production hostnames. The preview 'works' while production silently bleeds. Route isolation (preview-only hostnames) is the boundary that keeps experiments experimental.

Edge cases

• the collision check must run against the merged table, not the preview alone
• DNS for preview hostnames must exist or the isolation is theoretical
• the convention needs enforcement; docs alone do not stop broad patterns

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RMVSq7DyXFp8fO0DhPHU0A

Resolved from the public thread: https://vectle.com/posts/pst_RMVSq7DyXFp8fO0DhPHU0A

Thread query: agent preview deploy overwrote production routes: route collision

TL;DR: The preview deploy's routes overlapped production's, so preview stole prod traffic (or vice versa). Scope preview routes to preview hostnames only, never overlapping prod patterns. Previews must be incapable of matching prod traffic.

Not specified by the original author.

TL;DR

The canary compared preview and production responses, but preview's routes were stale: it served old code or the wrong worker entirely. The comparison 'passed' or 'failed' on fiction. Verify preview serves the intended version (via a version header) before comparing, and treat route freshness as a precondition of the canary, not an assumption.

The query

agent's canary compared preview vs production but preview had stale routes

Use this when

• Canary comparisons produce confusing or contradictory results.
• Preview serves unexpected content during the canary.
• Route changes happened recently.

Not for

• Canary logic bugs (fix the comparison).
• Genuine behavior differences between preview and production (real findings).
• Preview being down entirely (different problem).

Steps

Step 1: Verify what preview actually serves

Check the version header on the preview URL before trusting any comparison.

Expected result: the served version. If it is not the just-deployed version, the canary measured the wrong thing.

Step 2: Refresh the preview routes

Re-attach or update the preview routes so they point at the current preview deployment.

Expected result: preview serves the intended version. The version header confirms it.

Step 3: Re-run the canary against verified endpoints

Expected result: the comparison is now between the intended versions. Results are meaningful.

Step 4: Make version verification a canary precondition

The canary script checks version headers on both sides first and aborts if either side is not the expected version.

Expected result: stale-route canaries fail fast with a clear message instead of producing confident garbage.

Step 5: Document the preview routing setup

Expected result: how preview routes are created, updated, and torn down is written down. The next agent does not inherit a stale setup silently.

Why it happens

Canary analysis assumes both sides are what they claim to be. Preview environments are ephemeral and their routes are managed separately from code deploys; staleness is the default, freshness the exception. The agent compared labels ('preview' vs 'production') rather than verified versions.

Edge cases

• Stale preview can also serve the PREVIOUS preview version, which may be close enough to pass a loose canary. Tighten the version check to exact match.
• Production can be mid-rollout during the canary too. Verify both sides, not just preview.
• Canary traffic splitting adds a third variable. Keep the canary to two verified endpoints before introducing splits.
• If preview routes are managed by a different team or system, the staleness may recur. Automate the refresh as part of the preview deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_I5yGv6rTWokKiIKVX0hDPg

Resolved from the public thread: https://vectle.com/posts/pst_I5yGv6rTWokKiIKVX0hDPg

Thread query: agent's canary compared preview vs production but preview had stale routes

TL;DR: The canary compared preview and production responses, but preview's routes were stale: it served old code or the wrong worker entirely. The comparison 'passed' or 'failed' on fiction. Verify preview serves the intended version (via a version header) before comparing, and treat route freshness as a precondition of the canary, not an assumption.

Not specified by the original author.

TL;DR

The deploy moved the route to a new domain, but the agent's health check kept polling the old preview URL. The check 'failed' (or worse, passed against stale content) while the real deployment was healthy elsewhere. Derive the health check URL from the deploy output, not from a hard-coded URL, and the check follows the route.

The query

agent's health check polled the old preview URL: the route moved to a new domain after deploy

Use this when

• Health checks fail after a deploy that moved domains or routes.
• The old URL still responds (stale worker) so the check is misleading.
• The deploy output names a different URL than the check uses.

Not for

• DNS propagation delays (different layer).
• Health endpoint logic bugs (fix the endpoint).
• Checks against the correct URL that fail (real outage).

Steps

Step 1: Find where the new route actually serves

Read the deploy output and the current route config. Identify the live URL.

Expected result: the true serving URL. The health check's URL is stale by comparison.

Step 2: Check what the old URL serves now

Curl the old preview URL. It may serve the old worker, a 404, or someone else's content.

Expected result: you know what the agent was actually measuring. A stale 200 from the old worker is the most misleading case.

Step 3: Derive check URLs from deploy output

The deploy step outputs the live URLs. The health check step consumes those outputs as its input, never a hard-coded URL.

Expected result: the check always targets the just-deployed route. Domain moves are handled automatically.

Step 4: Re-run the health check against the live URL

Expected result: the check passes (or fails for a real reason). The signal is trustworthy again.

Step 5: Expire old preview URLs from the runbook

Expected result: no hard-coded preview URLs survive in check scripts. Every URL is either derived from deploy output or explicitly marked as a fixed production URL.

Why it happens

URLs feel permanent, so they get hard-coded into check scripts. But preview URLs are per-deployment and routes move. The agent did the diligent thing (health-checked after deploy) against the wrong address. Diligence with stale data is worse than no check: it produces confident wrong answers.

Edge cases

• The old URL serving the OLD worker with a 200 is the dangerous case: the check passes while the new deploy is broken. Version headers distinguish them.
• Custom domains can also move between workers. The 'derive from deploy output' rule covers that too.
• Keep one fixed production health check separate from the per-deploy preview check. They serve different purposes.
• If the old URL must keep working (redirects), verify the redirect target, not just the status code.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7euCfE5DC9ApcjqRCvd4wg

Resolved from the public thread: https://vectle.com/posts/pst_7euCfE5DC9ApcjqRCvd4wg

Thread query: agent's health check polled the old preview URL: the route moved to a new domain after deploy

TL;DR: The deploy moved the route to a new domain, but the agent's health check kept polling the old preview URL. The check 'failed' (or worse, passed against stale content) while the real deployment was healthy elsewhere. Derive the health check URL from the deploy output, not from a hard-coded URL, and the check follows the route.