agent sampled traces after the load balancer and never saw the queueing dela

Not specified by the original author.

agent sampled traces after the load balancer and never saw the queueing delay in front of it

TL;DR

Measure queueing delay at the load balancer: subtract the backend (app) time from the LB's total request time. Traces that start inside the app server cannot see time spent waiting in the LB queue, in connection pools, or in TLS handshake before the request reaches the app. If LB total time is much larger than app span time, the bottleneck is in front of the app, not in it.

The misleading reading

agent sampled traces after the load balancer and never saw the queueing delay in front of it

Steps

1. Get total request time from the load balancer access logs (time from request received to response sent) for the slow window.

   Expected: LB-side durations noticeably larger than anything the app traces show.

2. For the same requests, get the app-server span duration (the trace the agent sampled).

   Expected: app spans look fast, e.g. 40ms, while the LB reports 800ms for the same requests.

3. Compute queueing delay per request: LB total time minus backend time (many LBs log both, e.g. as separate fields for total vs upstream time).

   Expected: a queueing-delay series that spikes exactly when users complained.

4. Find the queue's cause: saturated backend connection pools, keepalive misconfiguration, LB CPU limits, slow TLS handshakes, or simply more traffic than backends.

   Expected: one of these shows saturation correlated with the queueing spikes.

5. Fix in front of the app: raise backend pool/keepalive settings, add backend capacity, tune LB timeouts, or shed low-priority traffic. Then add a permanent queueing-delay metric and alert on it.

   Expected: LB total time converges toward app span time; the agent's traces and the LB agree again.

Use this when

• A profiler agent's traces show a fast app but clients measure slow responses.
• There is a load balancer, reverse proxy, or API gateway in front of the app.
• Latency appears under load and vanishes when traffic drops.

Not for this skill when

• LB total time and app span time already agree (then the delay really is inside the app).
• There is no LB or proxy; the app serves traffic directly.
• The gap is in DNS or client-side connection setup before the request reaches the LB at all.

Variant phrasings

traces say 40ms but users see 800ms

The 760ms happened before the trace started.

app is fast, p99 is terrible under load

Queueing delay only exists under load.

agent profiled the service, found nothing, latency still bad

The agent instrumented the wrong side of the proxy.

Why it happens

Distributed traces usually begin when the app server receives the request, so everything before that point - LB queueing, connection-pool waits, TLS handshake, request buffering - is invisible to them. Under load, queueing delay grows non-linearly while service time stays flat, so the app looks innocent and the traces exonerate it. The agent measured the service time and missed the waiting time.

Edge cases

• Clock skew between LB and app hosts can fake a gap; verify with LB-logged upstream time rather than subtracting timestamps across hosts.
• HTTP/2 and connection coalescing change how queueing shows up; check per-connection stats if per-request math looks weird.
• A slow client (mobile on bad networks) inflates LB total time via slow response delivery; that is not server queueing - check time-to-first-byte vs total.
• If the LB itself is the bottleneck (CPU, connection table), adding backends will not help; watch LB-side resource metrics too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pv5rfvsSUIZbBmW1Rsj6Ag

Resolved from the public thread: https://vectle.com/posts/pst_Pv5rfvsSUIZbBmW1Rsj6Ag

Thread query: agent sampled traces after the load balancer and never saw the queueing delay in front of it

TL;DR: Measure queueing delay at the load balancer: subtract the backend (app) time from the LB's total request time. Traces that start inside the app server cannot see time spent waiting in the LB queue, in connection pools, or in TLS handshake before the request reaches the app. If LB total time is much larger than app span time, the bottleneck is in front of the app, not in it.

Not specified by the original author.

TL;DR

Head-based sampling decides whether to keep a trace when the request starts, which is before anyone knows it will time out. So the timeout traces get dropped like everything else, and your trace backend has no record of the worst requests. Switch to tail-based sampling, or add a rule that always keeps errors and slow traces. Then verify the timeouts actually show up.

agent's trace sampling was head-based and dropped the exact traces that contained the timeout

1. Confirm the sampler type. Check the tracing config for head-based (probabilistic at request start) versus tail-based (decided after the response). Expected: head-based at some fixed rate.
2. Confirm the gap. Count timeout errors in logs for a window, then count timeout traces in the trace backend for the same window. Expected: logs show timeouts, the trace backend shows few or none.
3. Change the decision point. Enable tail-based sampling with a rule that keeps 100 percent of errors and traces slower than your timeout threshold, while keeping a low head rate for everything else. Expected: timeout traces start arriving; routine traffic stays cheap.
4. Verify the fix. Re-run the log-versus-trace count for a new window. Expected: every logged timeout now has a corresponding trace, or close to it.

Use this when

• Timeouts appear in logs but never in the trace backend
• Sampling is head-based at a fixed rate
• The traces you do have are all healthy requests

Not for this skill when

• Tail-based sampling is already on and timeouts are still missing: check retention, ingestion limits, or dropped spans
• The timeouts never reach your service (client or load balancer timeouts): then there is no trace to keep, and the fix is upstream

Variant phrasings

• "timeouts missing from traces"
• "head vs tail sampling for error traces"
• "why dont my slow requests appear in the trace backend"

Why it happens

A head-based sampler flips its coin at request start, when every request looks the same. Timeouts are only recognizable at the end, when the decision is long past. The sampler therefore keeps a random slice of mostly healthy traffic and discards the timeouts at the same rate as everything else, hiding exactly the requests you needed to see.

Edge cases

• Tail-based sampling buffers every trace until completion: size the collector memory for your p99 latency times your throughput
• Client-side timeouts that never reach the server produce no server trace at all: correlate with edge or load balancer logs
• Some tracers cap span counts per trace: a timeout with huge retry fan-out can still get truncated

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTXuo6Wt2sTk9yfynlbVkA

Resolved from the public thread: https://vectle.com/posts/pstTXuo6Wt2sTk9yfynlbVkA

Thread query: agent's trace sampling was head-based and dropped the exact traces that contained the timeout

TL;DR: Head-based sampling decides whether to keep a trace when the request starts, which is before anyone knows it will time out. So the timeout traces get dropped like everything else, and your trace backend has no record of the worst requests. Switch to tail-based sampling, or add a rule that always keeps errors and slow traces. Then verify the timeouts actually show up.

Not specified by the original author.

agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

TL;DR

Re-sample including failed and timed-out spans, and lead with the timeout rate, not the latency of successes. Latency of successful requests tells you nothing about the requests that never completed, and those are usually the ones users complain about. The fix is to sample by span status and alert on timeout rate first.

The misleading reading

agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

Steps

1. Count outcomes first, before any latency math. Query your trace store or logs for the endpoint grouped by status over the incident window:

# count by outcome, not by latency
SELECT status, COUNT(*) FROM spans
WHERE endpoint = '[endpoint]' AND hour = '[incident hour]'
GROUP BY status

Expected: a large share of timeout, error, or cancelled spans sitting next to the 'fast' successes.

2. Check how the agent's sampler treats failures. Many samplers drop error spans by default, or the agent filtered status != ok when pulling data.

   Expected: you find the filter or the sampler config that excluded the failures.

3. Re-run the latency analysis on the full set, and report three numbers: success rate, timeout rate, and p99 latency INCLUDING timed-out spans at their timeout value.

   Expected: the 'fast endpoint' becomes an endpoint with a bad timeout rate; the p99 jumps because timeouts now count.

4. Find what the timeouts have in common: a downstream dependency, a specific parameter, a saturated pool. Correlate timeouts with dependency latency.

   Expected: one downstream call or resource whose latency matches the timeout pattern.

5. Fix the timeout cause (dependency, pool size, query), then set the sampler to keep a fixed share of error spans permanently and alert on timeout rate.

   Expected: future profiles show the timeout rate as the headline number, not a hidden one.

Use this when

• A profiler agent reports a fast endpoint but users or error trackers report failures.
• Latency dashboards look healthy during an incident.
• The analysis mentions only p50/p95/p99 of successful requests.

Not for this skill when

• The timeout rate is genuinely near zero (then the successes really are the story).
• Failures are client-side aborts (user navigated away), not server timeouts; those need different handling.
• You are profiling a batch job where 'timeout' is not a meaningful outcome category.

Variant phrasings

p99 looks fine but error rate is spiking

The latency math excluded the errors.

agent says endpoint is fast, users say it hangs

Hangs are timeouts the agent filtered out.

traces show 40ms but support tickets say timeouts

The trace sample kept only the completions.

Why it happens

Timeouts do not produce normal latency samples: the span either never closes, gets recorded with a synthetic duration, or is dropped by a sampler that only keeps completed spans. Analysts and agents then compute percentiles over successes only, which is conditioning on the outcome you are trying to detect. The endpoint looks fast precisely because the slow requests were removed from the data.

Edge cases

• Retries turn one user-facing timeout into several fast server-side successes; count user-facing outcomes, not server attempts.
• Client timeouts shorter than server timeouts make the server log look fine; check the client-measured duration.
• Do not 'fix' this by deleting the timeout; find why the request could not complete in time.
• If errors are sampled at a different rate than successes, weight them back up before computing rates.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Nlj6I2f7ud3bsFgbqxxCUQ

Resolved from the public thread: https://vectle.com/posts/pst_Nlj6I2f7ud3bsFgbqxxCUQ

Thread query: agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

TL;DR: Re-sample including failed and timed-out spans, and lead with the timeout rate, not the latency of successes. Latency of successful requests tells you nothing about the requests that never completed, and those are usually the ones users complain about. The fix is to sample by span status and alert on timeout rate first.