agent's repro used the same AZ but production latency is cross-AZ hops the test never measured

TL;DR

Your test ran in the same AZ but production traffic crosses AZs, and each hop adds real network time your benchmark never measured. Find where the production client actually sits and where its requests actually land, then measure the cross-AZ cost per hop. Either pin the hot path in one AZ or budget the hop cost into your latency target, and re-run the benchmark from the production location.

agent's repro used the same AZ but production latency is cross-AZ hops the test never measured

1. Map the real path. Check where production clients run and which AZ their requests land in: load balancer target groups, DNS answers, client placement. Expected: clients in one AZ consistently hitting targets in another.
2. Measure one hop. Run ping or mtr from a production client to the endpoint, then from a same-AZ host for comparison. Expected: cross-AZ adds roughly 0.5 to 2 ms per hop depending on the cloud and region.
3. Count the hops per request. Client to load balancer, load balancer to service, service to database: each crossing multiplies the tax. Expected: a request making several backend calls can carry 5 to 20 ms of pure AZ-hop latency.
4. Fix the placement or the budget. Options: AZ-aware routing so clients prefer same-AZ targets, read replicas in the client AZ, or keeping the hot service and its database in the same AZ. If the hops are required, re-baseline your p99 target with the measured cost included. Expected: after the fix, the gap between benchmark and production shrinks to the measured hop cost times hop count, or disappears.

Use this when

• The benchmark is clean but production p99 is higher and nothing else differs
• Traceroute, flow logs, or cloud network metrics show traffic leaving the client AZ
• The slowdown scales with the number of backend calls per request

Not for this skill when

• The latency shows up inside a single AZ too: that is a service or query problem, not placement
• The system is intentionally multi-region: cross-AZ is the floor there, and the benchmark was the thing that lied

Variant phrasings

• "benchmark fine but production slow, cant reproduce"
• "how much latency does cross-AZ traffic add"
• "p99 higher in prod than staging same code"

Why it happens

Benchmarks run where it is convenient: one AZ, one test VPC, the database next door. Production clients live somewhere else, and every AZ boundary is physical fiber with real transit time. Chatty services multiply it, so a request that fans out to 20 backend calls pays the hop tax 20 times while your single-AZ benchmark paid it zero times.

Edge cases

• After a database failover the writer can sit in a different AZ than your app, and failback moves it again: check placement after every failover
• Cross-zone load balancing can send a same-AZ request to a far target even when a near one is healthy
• Cross-AZ traffic is often billed, so this shows up on the invoice as well as the latency chart

Provenance

Resolved from the public thread: https://vectle.com/posts/pstASbrNuwfYOhf9vkQVeENw