agent summed service latencies instead of measuring the critical path

agent summed service latencies instead of measuring the critical path

TL;DR

Services call each other in parallel, so their latencies overlap; adding them double-counts the overlap. Read the critical path from the trace: the longest dependency chain. That is the number that matters, and it is always less than or equal to the sum.

agent summed service latencies instead of measuring the critical path

Steps

1. Open a representative trace and identify parallel spans: siblings under the same parent that overlap in time. Expected: You see the overlap the sum ignored.

2. Read the critical path: the longest chain of dependent spans from root to leaf. Most tracers highlight it. Expected: You have the true end-to-end latency and its components.

3. Compare the critical path to the naive sum. Expected: The sum is larger, sometimes dramatically.

4. Optimize the critical path, not the sum: a service off the critical path can be slow without affecting the user. Expected: Effort targets the chain that matters.

5. Teach the harness to report critical-path latency from traces instead of summing span durations. Expected: The metric is correct by construction.

When to use

• service latencies are summed into a total
• the summed total exceeds the measured request time
• prioritizing which service to optimize

When not to use

• all calls are strictly serial (then the sum is the critical path)
• you are adding costs, not latencies (costs do sum)
• measuring resource consumption rather than user-facing latency

Tool compatibility

• any distributed tracer with span timelines
• critical-path views

Variant phrasings

• "summed service latencies wrong"
• "critical path vs sum latency"
• "parallel spans double counted latency"

Why it happens

Latency adds only along dependencies. When service A calls B and C in parallel, the request waits for the slower of the two, not their sum. Adding span durations assumes a serial world that microservices deliberately avoid. The agent's total was always an overestimate, and optimizing by it misdirects effort to parallel branches.

Edge cases

• async fan-out with a join is the classic shape; the join waits for the slowest branch
• the critical path can shift between requests; look at the distribution, not one trace
• queueing time inside services is part of their span; the critical path includes it automatically

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WBP2sVmlxWVTk2cbcX0W4g