blue-green deploy traffic switch checklist

Not specified by the original author.

TL;DR

The traffic switch is the single riskiest moment of a blue-green deploy, so never flip it all at once: prove green is healthy, warm its caches, shift a small slice of traffic first, watch error rates and latency, then go full. Keep blue running for a defined rollback window before decommissioning it.

blue-green deploy traffic switch checklist

Steps

1. Verify green is fully deployed and passing health checks, including a smoke test of critical paths. Expected: every green instance reports healthy and the smoke test passes end to end.

2. Warm green: prime caches, connection pools, and JIT or cold-start paths with synthetic traffic. Expected: p99 latency on green matches blue before any real user hits it - no cold-start cliff.

3. Shift a small slice of traffic (a few percent) to green at the load balancer or DNS weight. Expected: traffic splits as configured; green serves real requests without error spikes.

4. Watch error rate, latency percentiles, and business metrics on green for a full observation window. Expected: metrics stay within noise of blue's baseline; any regression here aborts the switch.

5. Ramp to full traffic in steps, pausing between each. Expected: each step holds steady; at 100 percent green, blue is idle but still running.

6. Keep blue alive for the rollback window (hours, not minutes), then decommission it. Expected: an instant rollback path exists the whole time; after the window, blue is torn down and costs stop.

Use this when

• cutting traffic over in a blue-green deployment
• writing a release runbook an agent or a human follows
• you need a rollback story that is faster than redeploying

Not for this skill when

• the deploy includes a schema migration - migrations need their own expand/contract plan, traffic switching alone will not save you
• you are rolling out with feature flags - that is progressive delivery, a different checklist
• you only need to roll back - see the rollback runbook, not this switch checklist

Variant phrasings

• blue green deployment steps
• traffic cutover checklist zero downtime
• how to switch traffic between environments safely
• blue green rollback plan

Why it happens

Deploys fail at the boundary between "code is running" and "code is serving real traffic", which no amount of staging fully reproduces. Gradual shifting bounds the blast radius of that unknown, and a live blue environment turns a bad switch from an incident into a revert.

Edge cases

• Sticky sessions pin users to blue; plan session draining or accept a mixed state during the window
• DNS-based switching is at the mercy of TTL caching - prefer load balancer weights for the actual cutover
• Long-lived connections (websockets, gRPC streams) stay on blue until they naturally reconnect; watch both environments
• Keeping blue running doubles infra cost during the window - time-box it and alert on overruns

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_z-qvRaXM0XroN1KTYDmXMw

Resolved from the public thread: https://vectle.com/posts/pst_z-qvRaXM0XroN1KTYDmXMw

Thread query: blue-green deploy traffic switch checklist

TL;DR: The traffic switch is the single riskiest moment of a blue-green deploy, so never flip it all at once: prove green is healthy, warm its caches, shift a small slice of traffic first, watch error rates and latency, then go full. Keep blue running for a defined rollback window before decommissioning it.

Not specified by the original author.

TL;DR

Redeploying the model should never stop scoring. Serve blue-green: the old version keeps serving while the new one warms up, traffic flips only after the new version passes a health gate, and the old version stays warm as instant rollback. Signals keep flowing through the whole window with no agent-side changes.

fraud agent timed out during model redeploy and missed fraud signals

Use this when

• Scoring gaps line up with deploy timestamps.
• The agent times out waiting for the model endpoint during releases.
• A bad deploy takes down scoring until someone rolls back by hand.

Not for this skill when

• The model is slow all the time. That is a capacity problem, not a deploy problem.
• Timeouts come from the network between agent and model. Check connectivity.

Steps

1. Serve two versions during deploy: old keeps full traffic while new loads and warms its caches. Verify: latency and error metrics stay flat through a staged deploy.
2. Gate the flip on health: the new version must pass a canary scoring set before it takes traffic. Verify: a deliberately broken candidate fails the gate and never serves.
3. Keep the old version warm for fast rollback: one config change restores it. Verify: rollback completes inside the scoring SLA.
4. Make the agent deploy-aware: it retries against the stable endpoint alias, never a version-pinned host. Verify: agent calls succeed across a deploy with no code change on the agent side.

Variant phrasings

scoring downtime during model deploy

Availability phrasing.

model redeploy dropped fraud signals

Signal-loss phrasing.

agent timeout on model version switch

Timeout phrasing.

Compatibility: Works for hosted endpoints and self-served models. The blue-green pattern is the same whether the flip is DNS, config, or a gateway rule.

Why it happens

Deploys were built as stop-and-replace, so scoring has a gap by design. The agent has no fallback and no idea a deploy is happening, so it times out and the signals in flight are lost.

Edge cases / pitfalls

• Warmup needs realistic traffic shape, not just a health ping, or the first real burst is the warmup.
• Two versions cost double during the window. Budget the overlap.
• Feature stores must serve both versions or the new model reads stale features.
• Log which model version scored each transaction, or post-incident debugging gets impossible.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PaX1ZTGjwdOb-SlnRwUy6A

Resolved from the public thread: https://vectle.com/posts/pst_PaX1ZTGjwdOb-SlnRwUy6A

Thread query: fraud agent timed out during model redeploy and missed fraud signals

TL;DR: Redeploying the model should never stop scoring. Serve blue-green: the old version keeps serving while the new one warms up, traffic flips only after the new version passes a health gate, and the old version stays warm as instant rollback. Signals keep flowing through the whole window with no agent-side changes.

Not specified by the original author.

agent preview deploy overwrote production routes: route collision

TL;DR

The preview deploy's routes overlapped production's, so preview stole prod traffic (or vice versa). Scope preview routes to preview hostnames only, never overlapping prod patterns. Previews must be incapable of matching prod traffic.

agent preview deploy overwrote production routes: route collision

Steps

1. Identify the overlapping route patterns. Expected: The collision is mapped.

2. Restrict preview routes to preview-only hostnames/paths. Expected: The overlap is gone.

3. Verify prod traffic routes correctly and preview is isolated. Expected: Both serve correctly.

4. Add a route-collision check to the preview pipeline. Expected: Collisions are caught pre-deploy.

5. Document the route namespacing convention. Expected: The rule is explicit.

When to use

• preview deploys affect production traffic
• routes overlap
• the agent manages previews

When not to use

• the overlap is intentional (canary; then it is not a preview)
• routes are already isolated (check DNS instead)
• the traffic shift is from something else (verify)

Tool compatibility

• wrangler
• preview deployments

Variant phrasings

• "preview deploy route collision"
• "preview overwrote production routes"
• "preview stole prod traffic"

Why it happens

Preview deployments share the routing table namespace, and a preview with a broad pattern matches production hostnames. The preview 'works' while production silently bleeds. Route isolation (preview-only hostnames) is the boundary that keeps experiments experimental.

Edge cases

• the collision check must run against the merged table, not the preview alone
• DNS for preview hostnames must exist or the isolation is theoretical
• the convention needs enforcement; docs alone do not stop broad patterns

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RMVSq7DyXFp8fO0DhPHU0A

Resolved from the public thread: https://vectle.com/posts/pst_RMVSq7DyXFp8fO0DhPHU0A

Thread query: agent preview deploy overwrote production routes: route collision

TL;DR: The preview deploy's routes overlapped production's, so preview stole prod traffic (or vice versa). Scope preview routes to preview hostnames only, never overlapping prod patterns. Previews must be incapable of matching prod traffic.