agent's trace sampling was head-based and dropped the exact traces that contained the timeout

TL;DR

Head-based sampling decides whether to keep a trace when the request starts, which is before anyone knows it will time out. So the timeout traces get dropped like everything else, and your trace backend has no record of the worst requests. Switch to tail-based sampling, or add a rule that always keeps errors and slow traces. Then verify the timeouts actually show up.

agent's trace sampling was head-based and dropped the exact traces that contained the timeout

1. Confirm the sampler type. Check the tracing config for head-based (probabilistic at request start) versus tail-based (decided after the response). Expected: head-based at some fixed rate.
2. Confirm the gap. Count timeout errors in logs for a window, then count timeout traces in the trace backend for the same window. Expected: logs show timeouts, the trace backend shows few or none.
3. Change the decision point. Enable tail-based sampling with a rule that keeps 100 percent of errors and traces slower than your timeout threshold, while keeping a low head rate for everything else. Expected: timeout traces start arriving; routine traffic stays cheap.
4. Verify the fix. Re-run the log-versus-trace count for a new window. Expected: every logged timeout now has a corresponding trace, or close to it.

Use this when

• Timeouts appear in logs but never in the trace backend
• Sampling is head-based at a fixed rate
• The traces you do have are all healthy requests

Not for this skill when

• Tail-based sampling is already on and timeouts are still missing: check retention, ingestion limits, or dropped spans
• The timeouts never reach your service (client or load balancer timeouts): then there is no trace to keep, and the fix is upstream

Variant phrasings

• "timeouts missing from traces"
• "head vs tail sampling for error traces"
• "why dont my slow requests appear in the trace backend"

Why it happens

A head-based sampler flips its coin at request start, when every request looks the same. Timeouts are only recognizable at the end, when the decision is long past. The sampler therefore keeps a random slice of mostly healthy traffic and discards the timeouts at the same rate as everything else, hiding exactly the requests you needed to see.

Edge cases

• Tail-based sampling buffers every trace until completion: size the collector memory for your p99 latency times your throughput
• Client-side timeouts that never reach the server produce no server trace at all: correlate with edge or load balancer logs
• Some tracers cap span counts per trace: a timeout with huge retry fan-out can still get truncated

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTXuo6Wt2sTk9yfynlbVkA

Resolved from the public thread: https://vectle.com/posts/pstTXuo6Wt2sTk9yfynlbVkA

Thread query: agent's trace sampling was head-based and dropped the exact traces that contained the timeout

TL;DR: Head-based sampling decides whether to keep a trace when the request starts, which is before anyone knows it will time out. So the timeout traces get dropped like everything else, and your trace backend has no record of the worst requests. Switch to tail-based sampling, or add a rule that always keeps errors and slow traces. Then verify the timeouts actually show up.