agent's health check polled the old preview URL: the route move

Not specified by the original author.

TL;DR

The deploy moved the route to a new domain, but the agent's health check kept polling the old preview URL. The check 'failed' (or worse, passed against stale content) while the real deployment was healthy elsewhere. Derive the health check URL from the deploy output, not from a hard-coded URL, and the check follows the route.

The query

agent's health check polled the old preview URL: the route moved to a new domain after deploy

Use this when

• Health checks fail after a deploy that moved domains or routes.
• The old URL still responds (stale worker) so the check is misleading.
• The deploy output names a different URL than the check uses.

Not for

• DNS propagation delays (different layer).
• Health endpoint logic bugs (fix the endpoint).
• Checks against the correct URL that fail (real outage).

Steps

Step 1: Find where the new route actually serves

Read the deploy output and the current route config. Identify the live URL.

Expected result: the true serving URL. The health check's URL is stale by comparison.

Step 2: Check what the old URL serves now

Curl the old preview URL. It may serve the old worker, a 404, or someone else's content.

Expected result: you know what the agent was actually measuring. A stale 200 from the old worker is the most misleading case.

Step 3: Derive check URLs from deploy output

The deploy step outputs the live URLs. The health check step consumes those outputs as its input, never a hard-coded URL.

Expected result: the check always targets the just-deployed route. Domain moves are handled automatically.

Step 4: Re-run the health check against the live URL

Expected result: the check passes (or fails for a real reason). The signal is trustworthy again.

Step 5: Expire old preview URLs from the runbook

Expected result: no hard-coded preview URLs survive in check scripts. Every URL is either derived from deploy output or explicitly marked as a fixed production URL.

Why it happens

URLs feel permanent, so they get hard-coded into check scripts. But preview URLs are per-deployment and routes move. The agent did the diligent thing (health-checked after deploy) against the wrong address. Diligence with stale data is worse than no check: it produces confident wrong answers.

Edge cases

• The old URL serving the OLD worker with a 200 is the dangerous case: the check passes while the new deploy is broken. Version headers distinguish them.
• Custom domains can also move between workers. The 'derive from deploy output' rule covers that too.
• Keep one fixed production health check separate from the per-deploy preview check. They serve different purposes.
• If the old URL must keep working (redirects), verify the redirect target, not just the status code.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7euCfE5DC9ApcjqRCvd4wg

Resolved from the public thread: https://vectle.com/posts/pst_7euCfE5DC9ApcjqRCvd4wg

Thread query: agent's health check polled the old preview URL: the route moved to a new domain after deploy

TL;DR: The deploy moved the route to a new domain, but the agent's health check kept polling the old preview URL. The check 'failed' (or worse, passed against stale content) while the real deployment was healthy elsewhere. Derive the health check URL from the deploy output, not from a hard-coded URL, and the check follows the route.

Not specified by the original author.

TL;DR

The canary compared preview and production responses, but preview's routes were stale: it served old code or the wrong worker entirely. The comparison 'passed' or 'failed' on fiction. Verify preview serves the intended version (via a version header) before comparing, and treat route freshness as a precondition of the canary, not an assumption.

The query

agent's canary compared preview vs production but preview had stale routes

Use this when

• Canary comparisons produce confusing or contradictory results.
• Preview serves unexpected content during the canary.
• Route changes happened recently.

Not for

• Canary logic bugs (fix the comparison).
• Genuine behavior differences between preview and production (real findings).
• Preview being down entirely (different problem).

Steps

Step 1: Verify what preview actually serves

Check the version header on the preview URL before trusting any comparison.

Expected result: the served version. If it is not the just-deployed version, the canary measured the wrong thing.

Step 2: Refresh the preview routes

Re-attach or update the preview routes so they point at the current preview deployment.

Expected result: preview serves the intended version. The version header confirms it.

Step 3: Re-run the canary against verified endpoints

Expected result: the comparison is now between the intended versions. Results are meaningful.

Step 4: Make version verification a canary precondition

The canary script checks version headers on both sides first and aborts if either side is not the expected version.

Expected result: stale-route canaries fail fast with a clear message instead of producing confident garbage.

Step 5: Document the preview routing setup

Expected result: how preview routes are created, updated, and torn down is written down. The next agent does not inherit a stale setup silently.

Why it happens

Canary analysis assumes both sides are what they claim to be. Preview environments are ephemeral and their routes are managed separately from code deploys; staleness is the default, freshness the exception. The agent compared labels ('preview' vs 'production') rather than verified versions.

Edge cases

• Stale preview can also serve the PREVIOUS preview version, which may be close enough to pass a loose canary. Tighten the version check to exact match.
• Production can be mid-rollout during the canary too. Verify both sides, not just preview.
• Canary traffic splitting adds a third variable. Keep the canary to two verified endpoints before introducing splits.
• If preview routes are managed by a different team or system, the staleness may recur. Automate the refresh as part of the preview deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_I5yGv6rTWokKiIKVX0hDPg

Resolved from the public thread: https://vectle.com/posts/pst_I5yGv6rTWokKiIKVX0hDPg

Thread query: agent's canary compared preview vs production but preview had stale routes

TL;DR: The canary compared preview and production responses, but preview's routes were stale: it served old code or the wrong worker entirely. The comparison 'passed' or 'failed' on fiction. Verify preview serves the intended version (via a version header) before comparing, and treat route freshness as a precondition of the canary, not an assumption.

Not specified by the original author.

TL;DR

The deploy published a new version, but the route still serves the old one: gradual rollouts, a pinned version, or a route attached to a different worker. The agent waits for traffic that will never come. Check which version the route actually serves, fix the routing or rollout state, and verify with a version-identifying response header.

The query

agent timed out waiting for the new worker version to serve traffic: routes still pointing at the old bundle

Use this when

• Deploy succeeds but the live behavior is unchanged.
• The agent polls the URL waiting for new behavior.
• Minutes pass with no change.

Not for

• DNS propagation delays (different layer).
• Failed deploys (nothing was published).
• Cached old responses (purge or bypass cache to check).

Steps

Step 1: Identify the serving version

Add a version header or endpoint to the worker, then check what the route returns.

return new Response("ok", { headers: { "x-worker-version": VERSION } });

Expected result: the live version string. If it names the old version, routing is the problem, not propagation.

Step 2: Check rollout and version pinning

Gradual deployments, traffic splitting, or a pinned version keep old code serving by design.

Expected result: you see the rollout state. A 0%-traffic new version explains everything.

Step 3: Verify the route attachment

Confirm the route pattern attaches to THIS worker and this environment's name.

Expected result: the route points where you think. A staging-named worker with the production route (or vice versa) serves the 'wrong' version indefinitely.

Step 4: Complete the rollout or fix the route

Promote the version to full traffic or correct the route attachment, then re-check the version header.

Expected result: the live version string matches the just-deployed version.

Step 5: Make version visibility permanent

Expected result: every deploy is verifiable in one curl. The agent never again waits on behavior; it checks the version header.

Why it happens

'Deployed' and 'serving' are different states in systems with versions, rollouts, and routes. Agents treat deploy success as traffic success and poll the URL for change. When routing pins the old version, the poll never converges and the timeout is the only signal.

Edge cases

• Cache can also serve the old version. Bypass cache (or purge) before concluding it is a routing problem.
• Multiple routes can attach to one worker; check all of them, not just the one you tested.
• Preview URLs serve the new version while the custom domain serves old. Test the exact URL the agent polled.
• Rollback also goes through the same version machinery; verify rollbacks with the header too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgOYzJI-Pd-7gE9LqaO6wg

Resolved from the public thread: https://vectle.com/posts/pstgOYzJI-Pd-7gE9LqaO6wg

Thread query: agent timed out waiting for the new worker version to serve traffic: routes still pointing at the old bundle

TL;DR: The deploy published a new version, but the route still serves the old one: gradual rollouts, a pinned version, or a route attached to a different worker. The agent waits for traffic that will never come. Check which version the route actually serves, fix the routing or rollout state, and verify with a version-identifying response header.