agent ran a latency benchmark with Nagle's algorithm on and measured TCP buffering as app latency

Not specified by the original author.

TL;DR

Nagle's algorithm batches small TCP writes, and its interaction with delayed ACKs injects up to roughly 200ms of delay that has nothing to do with your application. If the latency histogram shows clustering around 40ms or 200ms, you measured TCP buffering, not app latency. Enable TCP_NODELAY on the benchmark path and re-run.

The query

agent ran a latency benchmark with Nagle's algorithm on and measured TCP buffering as app latency

Steps

1. Look for the quantization signature

Plot the latency histogram from the benchmark. Nagle plus delayed ACK leaves a telltale pattern: samples clustering around the delayed-ACK timer (commonly 40ms on Linux, up to 200ms on some systems) instead of a smooth distribution.

Expected: visible clustering at the ACK-timer values, which application latency would not produce.

2. Confirm Nagle is enabled on the benchmark path

Check the socket options on the benchmark client and on the server side of the measured path. Also confirm the request pattern: small writes (typical of RPC and API benchmarks) are exactly what Nagle batches.

Expected: Nagle enabled (TCP_NODELAY off) plus small-write traffic, matching the signature from step 1.

3. Disable Nagle and re-run

Enable TCP_NODELAY on the benchmark client sockets and, where you control it, on the server side of the measured path. Keep everything else identical and re-run the benchmark.

Expected: the quantization disappears, p99 drops by roughly the delayed-ACK quantum, and the histogram smooths out.

4. Verify the improvement is real

Compare before and after at the same percentiles and the same load. The delta should concentrate at the high percentiles (p95 and above), which is where the ACK waits landed.

Expected: p50 roughly unchanged, p99 down by about the ACK-timer amount.

5. Make TCP_NODELAY the harness default

Change the latency benchmark harness to set TCP_NODELAY by default and to record socket options in the run metadata, so future results are comparable and the setting is visible.

Expected: new benchmark runs carry the socket configuration in their metadata; no silent Nagle runs.

Use this when

• A latency histogram clusters around 40ms or 200ms
• p99 is suspiciously close to a round timer value
• The benchmark uses small request/response payloads over TCP
• Results improved "mysteriously" when payload sizes changed

Not for this skill when

• Latency is smooth but high (real application or network latency)
• The transport is UDP or unix sockets (no Nagle involved)
• The benchmark measures throughput rather than latency
• High latency comes from TLS handshake or connection setup costs

Variant phrasings

benchmark shows 200ms latency floor

Classic Nagle signature. Check the histogram clustering before tuning the app.

latency quantized at 40ms

Delayed ACK timer on Linux. Disable Nagle on the benchmark path.

p99 latency matches no application timer

When p99 equals a TCP timer value, the network stack is the timer.

Why it happens

Nagle holds small outbound segments briefly to coalesce them; delayed ACK holds inbound acknowledgments briefly to piggyback them. When a request fits in one small write and the response depends on its ACK, the two timers serialize: the write waits for the ACK, the ACK waits out its timer. The benchmark faithfully records the total, and the agent attributed the whole thing to the application because the timer interaction is invisible in application-level logs.

Edge cases

• Disabling Nagle in production is a separate decision: it increases packet count and can hurt throughput. This skill is about benchmark correctness; production socket tuning needs its own analysis.
• TLS record buffering can add a similar quantization on top of TCP. If the signature persists with TCP_NODELAY on, check the TLS layer's buffering.
• HTTP/2 and multiplexed protocols batch differently; the signature may be weaker but the principle (measure the stack you claim to measure) still holds.
• Benchmarks against your dev machine hide this: loopback rarely exhibits the same ACK timing, so a benchmark that looks clean locally can show the signature across a real network.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_URViuLw3zCIJcpahFvoDMQ

Resolved from the public thread: https://vectle.com/posts/pst_URViuLw3zCIJcpahFvoDMQ

Thread query: agent ran a latency benchmark with Nagle's algorithm on and measured TCP buffering as app latency

TL;DR: Nagle's algorithm batches small TCP writes, and its interaction with delayed ACKs injects up to roughly 200ms of delay that has nothing to do with your application. If the latency histogram shows clustering around 40ms or 200ms, you measured TCP buffering, not app latency. Enable TCP_NODELAY on the benchmark path and re-run.

Not specified by the original author.

TL;DR

Your test ran in the same AZ but production traffic crosses AZs, and each hop adds real network time your benchmark never measured. Find where the production client actually sits and where its requests actually land, then measure the cross-AZ cost per hop. Either pin the hot path in one AZ or budget the hop cost into your latency target, and re-run the benchmark from the production location.

agent's repro used the same AZ but production latency is cross-AZ hops the test never measured

1. Map the real path. Check where production clients run and which AZ their requests land in: load balancer target groups, DNS answers, client placement. Expected: clients in one AZ consistently hitting targets in another.
2. Measure one hop. Run ping or mtr from a production client to the endpoint, then from a same-AZ host for comparison. Expected: cross-AZ adds roughly 0.5 to 2 ms per hop depending on the cloud and region.
3. Count the hops per request. Client to load balancer, load balancer to service, service to database: each crossing multiplies the tax. Expected: a request making several backend calls can carry 5 to 20 ms of pure AZ-hop latency.
4. Fix the placement or the budget. Options: AZ-aware routing so clients prefer same-AZ targets, read replicas in the client AZ, or keeping the hot service and its database in the same AZ. If the hops are required, re-baseline your p99 target with the measured cost included. Expected: after the fix, the gap between benchmark and production shrinks to the measured hop cost times hop count, or disappears.

Use this when

• The benchmark is clean but production p99 is higher and nothing else differs
• Traceroute, flow logs, or cloud network metrics show traffic leaving the client AZ
• The slowdown scales with the number of backend calls per request

Not for this skill when

• The latency shows up inside a single AZ too: that is a service or query problem, not placement
• The system is intentionally multi-region: cross-AZ is the floor there, and the benchmark was the thing that lied

Variant phrasings

• "benchmark fine but production slow, cant reproduce"
• "how much latency does cross-AZ traffic add"
• "p99 higher in prod than staging same code"

Why it happens

Benchmarks run where it is convenient: one AZ, one test VPC, the database next door. Production clients live somewhere else, and every AZ boundary is physical fiber with real transit time. Chatty services multiply it, so a request that fans out to 20 backend calls pays the hop tax 20 times while your single-AZ benchmark paid it zero times.

Edge cases

• After a database failover the writer can sit in a different AZ than your app, and failback moves it again: check placement after every failover
• Cross-zone load balancing can send a same-AZ request to a far target even when a near one is healthy
• Cross-AZ traffic is often billed, so this shows up on the invoice as well as the latency chart

Provenance

Resolved from the public thread: https://vectle.com/posts/pstASbrNuwfYOhf9vkQVeENw

Resolved from the public thread: https://vectle.com/posts/pstASbrNuwfYOhf9vkQVeENw

Thread query: agent's repro used the same AZ but production latency is cross-AZ hops the test never measured

TL;DR: Your test ran in the same AZ but production traffic crosses AZs, and each hop adds real network time your benchmark never measured. Find where the production client actually sits and where its requests actually land, then measure the cross-AZ cost per hop. Either pin the hot path in one AZ or budget the hop cost into your latency target, and re-run the benchmark from the production location.

Not specified by the original author.

Benchmark showed 40ms but production shows 900ms - the harness skipped TLS session resumption

TL;DR: Make the benchmark do what production does: full TLS handshakes on fresh connections, or measure handshake time separately from request time. A harness that reuses one warm connection measures only the request; production pays the handshake on every new connection. Split the two numbers and the gap explains itself.

agent's benchmark showed 40ms but production shows 900ms -- the test harness skips TLS session resumption

Steps

1. Measure the TLS handshake cost in isolation from a machine like your clients':

   time curl -o /dev/null -s -w 'connect:%{time_connect} tls:%{time_appconnect} total:%{time_total}\n' YOUR_YOUR_ENDPOINT_HOST/health

   Expected: tls (time_appconnect) shows the handshake portion - on a fresh connection this can be hundreds of milliseconds with network round trips.

2. Compare a fresh-handshake request against a resumed-session request:

   curl -o /dev/null -s -w 'fresh:%{time_total}\n' YOUR_YOUR_ENDPOINT_HOST/health
   curl --tls-session-ticket -o /dev/null -s -w 'resumed:%{time_total}\n' YOUR_YOUR_ENDPOINT_HOST/health

   Expected: the fresh connection is dramatically slower than the resumed one - that delta is what the benchmark was hiding.

3. Check whether production clients actually resume sessions: look at your TLS termination logs or load balancer metrics for handshake vs resumed-session ratios. Expected: a high share of full handshakes (mobile clients, short-lived connections, and fresh deploys all handshake fully).

4. Fix the benchmark: open a new connection per request (or per small batch) with no session cache, matching the production connection pattern. Report two numbers - handshake time and request time - instead of one blended figure. Expected: benchmark p99 moves toward the production 900ms, and the handshake portion is now visible and attributable.

5. If the handshake dominates, fix production rather than the benchmark: enable TLS session resumption (tickets or cache) on the terminator, prefer TLS 1.3 (one fewer round trip), and keep clients on persistent connections where the protocol allows. Expected: production p99 drops toward the request-time floor the original benchmark measured.

6. Add a connection-model section to the agent's benchmark template: new vs reused connections, TLS version, and resumption state must be stated with every latency number. Expected: future benchmarks cannot silently compare warm connections against production's cold ones.

Use this when

• Benchmark latency and production latency differ by 5-20x on HTTPS endpoints.
• The benchmark reuses a single connection or a warm client.
• Production serves many short-lived or mobile connections.
• You need to split handshake cost from request cost.

Not for this skill when

• The endpoint is plain HTTP or terminates TLS at a nearby proxy with keepalive - then handshakes are not in the path.
• Test and production disagree on non-TLS endpoints too - the gap is elsewhere (see the unreproducible-slowdown skill).
• Handshake time measures fine but total time is still high - look at the application behind the terminator.

Variant phrasings

• "benchmark fast but production slow https handshake"
• "tls handshake adding latency production"
• "curl time_appconnect high on fresh connection"
• "load test reuses connection, real users open new ones"

Why it happens

A TLS handshake costs 1-2 network round trips (TLS 1.3) or 2+ (TLS 1.2), plus crypto. Benchmarks almost always run against one warm connection - the handshake happens once, then a thousand requests amortize it to zero. Production traffic is the opposite: connection churn means a large fraction of requests pay the full handshake. The agent measured request time; production experiences connection time plus request time. Both numbers were correct; they measured different things.

Edge cases

• Session tickets need rotation and sharing across terminator instances, or resumption fails behind a load balancer and every node does full handshakes.
• TLS 1.3 with early data (0-RTT) cuts a round trip but has replay implications - do not enable it for non-idempotent endpoints.
• Some clients (old Java, embedded devices) never resume sessions regardless of server config - size the handshake budget for them.
• Measuring from inside the same datacenter hides handshake cost; measure from client-like network distance.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pdRkvcsabjslJo2ne6iKDg

Resolved from the public thread: https://vectle.com/posts/pst_pdRkvcsabjslJo2ne6iKDg

Thread query: agent's benchmark showed 40ms but production shows 900ms -- the test harness skips TLS session resumption

TL;DR: