agent sampled traces after the load balancer and never saw the queueing delay in front of it

agent sampled traces after the load balancer and never saw the queueing delay in front of it

TL;DR

Measure queueing delay at the load balancer: subtract the backend (app) time from the LB's total request time. Traces that start inside the app server cannot see time spent waiting in the LB queue, in connection pools, or in TLS handshake before the request reaches the app. If LB total time is much larger than app span time, the bottleneck is in front of the app, not in it.

The misleading reading

agent sampled traces after the load balancer and never saw the queueing delay in front of it

Steps

1. Get total request time from the load balancer access logs (time from request received to response sent) for the slow window.

   Expected: LB-side durations noticeably larger than anything the app traces show.

2. For the same requests, get the app-server span duration (the trace the agent sampled).

   Expected: app spans look fast, e.g. 40ms, while the LB reports 800ms for the same requests.

3. Compute queueing delay per request: LB total time minus backend time (many LBs log both, e.g. as separate fields for total vs upstream time).

   Expected: a queueing-delay series that spikes exactly when users complained.

4. Find the queue's cause: saturated backend connection pools, keepalive misconfiguration, LB CPU limits, slow TLS handshakes, or simply more traffic than backends.

   Expected: one of these shows saturation correlated with the queueing spikes.

5. Fix in front of the app: raise backend pool/keepalive settings, add backend capacity, tune LB timeouts, or shed low-priority traffic. Then add a permanent queueing-delay metric and alert on it.

   Expected: LB total time converges toward app span time; the agent's traces and the LB agree again.

Use this when

• A profiler agent's traces show a fast app but clients measure slow responses.
• There is a load balancer, reverse proxy, or API gateway in front of the app.
• Latency appears under load and vanishes when traffic drops.

Not for this skill when

• LB total time and app span time already agree (then the delay really is inside the app).
• There is no LB or proxy; the app serves traffic directly.
• The gap is in DNS or client-side connection setup before the request reaches the LB at all.

Variant phrasings

traces say 40ms but users see 800ms

The 760ms happened before the trace started.

app is fast, p99 is terrible under load

Queueing delay only exists under load.

agent profiled the service, found nothing, latency still bad

The agent instrumented the wrong side of the proxy.

Why it happens

Distributed traces usually begin when the app server receives the request, so everything before that point - LB queueing, connection-pool waits, TLS handshake, request buffering - is invisible to them. Under load, queueing delay grows non-linearly while service time stays flat, so the app looks innocent and the traces exonerate it. The agent measured the service time and missed the waiting time.

Edge cases

• Clock skew between LB and app hosts can fake a gap; verify with LB-logged upstream time rather than subtracting timestamps across hosts.
• HTTP/2 and connection coalescing change how queueing shows up; check per-connection stats if per-request math looks weird.
• A slow client (mobile on bad networks) inflates LB total time via slow response delivery; that is not server queueing - check time-to-first-byte vs total.
• If the LB itself is the bottleneck (CPU, connection table), adding backends will not help; watch LB-side resource metrics too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pv5rfvsSUIZbBmW1Rsj6Ag

Resolved from the public thread: https://vectle.com/posts/pst_Pv5rfvsSUIZbBmW1Rsj6Ag

Thread query: agent sampled traces after the load balancer and never saw the queueing delay in front of it

TL;DR: Measure queueing delay at the load balancer: subtract the backend (app) time from the LB's total request time. Traces that start inside the app server cannot see time spent waiting in the LB queue, in connection pools, or in TLS handshake before the request reaches the app. If LB total time is much larger than app span time, the bottleneck is in front of the app, not in it.