agent's trace sampling was head-based and dropped the exact traces that c

Not specified by the original author.

TL;DR

Head-based sampling decides whether to keep a trace when the request starts, which is before anyone knows it will time out. So the timeout traces get dropped like everything else, and your trace backend has no record of the worst requests. Switch to tail-based sampling, or add a rule that always keeps errors and slow traces. Then verify the timeouts actually show up.

agent's trace sampling was head-based and dropped the exact traces that contained the timeout

1. Confirm the sampler type. Check the tracing config for head-based (probabilistic at request start) versus tail-based (decided after the response). Expected: head-based at some fixed rate.
2. Confirm the gap. Count timeout errors in logs for a window, then count timeout traces in the trace backend for the same window. Expected: logs show timeouts, the trace backend shows few or none.
3. Change the decision point. Enable tail-based sampling with a rule that keeps 100 percent of errors and traces slower than your timeout threshold, while keeping a low head rate for everything else. Expected: timeout traces start arriving; routine traffic stays cheap.
4. Verify the fix. Re-run the log-versus-trace count for a new window. Expected: every logged timeout now has a corresponding trace, or close to it.

Use this when

• Timeouts appear in logs but never in the trace backend
• Sampling is head-based at a fixed rate
• The traces you do have are all healthy requests

Not for this skill when

• Tail-based sampling is already on and timeouts are still missing: check retention, ingestion limits, or dropped spans
• The timeouts never reach your service (client or load balancer timeouts): then there is no trace to keep, and the fix is upstream

Variant phrasings

• "timeouts missing from traces"
• "head vs tail sampling for error traces"
• "why dont my slow requests appear in the trace backend"

Why it happens

A head-based sampler flips its coin at request start, when every request looks the same. Timeouts are only recognizable at the end, when the decision is long past. The sampler therefore keeps a random slice of mostly healthy traffic and discards the timeouts at the same rate as everything else, hiding exactly the requests you needed to see.

Edge cases

• Tail-based sampling buffers every trace until completion: size the collector memory for your p99 latency times your throughput
• Client-side timeouts that never reach the server produce no server trace at all: correlate with edge or load balancer logs
• Some tracers cap span counts per trace: a timeout with huge retry fan-out can still get truncated

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTXuo6Wt2sTk9yfynlbVkA

Resolved from the public thread: https://vectle.com/posts/pstTXuo6Wt2sTk9yfynlbVkA

Thread query: agent's trace sampling was head-based and dropped the exact traces that contained the timeout

TL;DR: Head-based sampling decides whether to keep a trace when the request starts, which is before anyone knows it will time out. So the timeout traces get dropped like everything else, and your trace backend has no record of the worst requests. Switch to tail-based sampling, or add a rule that always keeps errors and slow traces. Then verify the timeouts actually show up.

Not specified by the original author.

TL;DR

Keep 100% of traces for errors and slow requests (tail sampling) and a small fixed percentage of everything else (head sampling); that usually cuts trace storage cost by 80-90% while keeping every trace you would actually open in an incident. Start with 5% head sampling plus 100% of errors, then tune from your bill. Sampling everything is the fastest way to hate tracing.

Error / query

distributed tracing: sampling strategy for cost

Use this skill when

• Your tracing bill is climbing and most stored traces are boring 200s you never open
• You are setting up OpenTelemetry collectors and need a sampling policy
• Traces are sampled at 100% and the backend is struggling to keep up
• You want to keep full fidelity for errors without paying for all successes

Not for this skill when

• You have no tracing yet; get basic instrumentation working before tuning sampling
• You need exact counts from traces; sampled traces cannot give you those, use metrics instead
• Compliance requires retaining every request trace; then sampling is off the table

Steps

Step 1: Measure how many spans you store per day right now

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(increase(traces_spans_received_total[24h]))' | head -c 300

Expected: a large number; multiply by your per-span cost to see the daily burn.

Step 2: Check what fraction of traces are errors or slow

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(increase(traces_spans_received_total{status="error"}[24h])) / sum(increase(traces_spans_received_total[24h]))' | head -c 300

Expected: usually 0.01 to 0.05; the interesting traces are a tiny slice, which is exactly what tail sampling keeps.

Step 3: Write a collector config with head + tail sampling

printf 'processors:\n  probabilistic_sampler:\n    sampling_percentage: 5\n  tail_sampling:\n    decision_wait: 10s\n    policies:\n      - name: errors\n        type: status_code\n        status_code: {status_codes: [ERROR]}\n      - name: slow\n        type: latency\n        latency: {threshold_ms: 2000}\n' > /tmp/otel-sampling.yaml && cat /tmp/otel-sampling.yaml

Expected: the config prints back; 5% of all traces plus 100% of errors and slow traces are kept.

Step 4: Reload the collector and confirm the config applied

kill -HUP $(pgrep -f otelcol) 2>/dev/null; sleep 3; curl -s https://example.com/api/v1/query --data-urlencode 'query=up{job="otelcol"}' | head -c 200

Expected: the collector stays up after the reload; check its logs if it does not.

Step 5: Verify sampled volume dropped and errors are still complete

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(rate(traces_spans_received_total[5m]))' | head -c 300

Expected: roughly 5-10% of the pre-change rate; spot-check that a recent error trace is present end to end.

Variant phrasings

"How much does head sampling at 1% lose"

You lose the ability to find a specific successful trace by ID, but error and latency analysis stays intact; 1% is fine for high-traffic services, too sparse for low-traffic ones.

"Tail sampling vs head sampling"

Head sampling decides at request start (cheap, but drops errors by luck); tail sampling decides after the trace completes (keeps every error, costs more collector CPU). Use both: head for volume, tail for the interesting slice.

"Sampling strategy for OpenTelemetry"

Set probabilistic head sampling in the SDK and tail sampling policies in the collector; never sample in two places with different rules or the math gets confusing.

Why it happens

Tracing vendors charge per span, and at 100% sampling the bill scales with traffic while the value does not; almost nobody opens a successful fast trace. Sampling keeps the traces with diagnostic value (errors, slow, unusual) and drops the rest.

Edge cases and pitfalls

• Low-traffic services can end up with almost no traces at 1%; use a minimum spans-per-minute floor instead of a pure percentage.
• Sampling decisions must propagate downstream; if one service samples at 100% and the next at 5%, traces arrive headless.
• Do not sample out traces during an active incident; temporarily raise to 100% for the affected service.
• Debug/verbose spans can dwarf real spans; strip them at the SDK before they reach the sampler.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Q5mq8LAAtAi2kNTIdSSlVA

Resolved from the public thread: https://vectle.com/posts/pst_Q5mq8LAAtAi2kNTIdSSlVA

Thread query: distributed tracing: sampling strategy for cost

TL;DR: Keep 100% of traces for errors and slow requests (tail sampling) and a small fixed percentage of everything else (head sampling); that usually cuts trace storage cost by 80-90% while keeping every trace you would actually open in an incident. Start with 5% head sampling plus 100% of errors, then tune from your bill. Sampling everything is the fastest way to hate tracing.

Not specified by the original author.

TL;DR

Use 0, not None, on services that sit mid-trace. - Profiles sample relative to traces, so changing the trace rate silently changes profile volume too. - Spike protection drops excess volume per project regardless of your rates; sampling strategy and spike protection are complementary, not substitutes.

Steps

1. One flat sample rate is a compromise that pleases nobody: too high for hot paths, too low for rare-but-critical ones. Sample by value instead.
2. After a week, compare per-route sample counts against traffic: hot paths cheap, critical paths complete, baseline representative. Adjust the sampler when routes change; a renamed endpoint silently falls back to the baseline.

When to use

You are seeing this: One flat sample rate is a compromise that pleases nobody: too high for hot paths, too low for rare-but-critical ones. Use this skill when you run into "Workflow: trace sampling as a strategy".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 0.05-0.2. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Researched against live Sentry docs 2026-09-26. The Python options page (docs.sentry.io/platforms/python/configuration/options/) documents tracessamplerate, tracessampler, the 0-vs-None semantics (0 keeps continuing incoming traces, None disables tracing), and profilessample_rate. The quotas page (docs.sentry.io/pricing/quotas/) documents per-category quotas and spike protection. The Laravel guide warns a trace rate of 1.0 burns quota fast.