distributed tracing: sampling strategy for cost

TL;DR

Keep 100% of traces for errors and slow requests (tail sampling) and a small fixed percentage of everything else (head sampling); that usually cuts trace storage cost by 80-90% while keeping every trace you would actually open in an incident. Start with 5% head sampling plus 100% of errors, then tune from your bill. Sampling everything is the fastest way to hate tracing.

Error / query

distributed tracing: sampling strategy for cost

Use this skill when

• Your tracing bill is climbing and most stored traces are boring 200s you never open
• You are setting up OpenTelemetry collectors and need a sampling policy
• Traces are sampled at 100% and the backend is struggling to keep up
• You want to keep full fidelity for errors without paying for all successes

Not for this skill when

• You have no tracing yet; get basic instrumentation working before tuning sampling
• You need exact counts from traces; sampled traces cannot give you those, use metrics instead
• Compliance requires retaining every request trace; then sampling is off the table

Steps

Step 1: Measure how many spans you store per day right now

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(increase(traces_spans_received_total[24h]))' | head -c 300

Expected: a large number; multiply by your per-span cost to see the daily burn.

Step 2: Check what fraction of traces are errors or slow

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(increase(traces_spans_received_total{status="error"}[24h])) / sum(increase(traces_spans_received_total[24h]))' | head -c 300

Expected: usually 0.01 to 0.05; the interesting traces are a tiny slice, which is exactly what tail sampling keeps.

Step 3: Write a collector config with head + tail sampling

printf 'processors:\n  probabilistic_sampler:\n    sampling_percentage: 5\n  tail_sampling:\n    decision_wait: 10s\n    policies:\n      - name: errors\n        type: status_code\n        status_code: {status_codes: [ERROR]}\n      - name: slow\n        type: latency\n        latency: {threshold_ms: 2000}\n' > /tmp/otel-sampling.yaml && cat /tmp/otel-sampling.yaml

Expected: the config prints back; 5% of all traces plus 100% of errors and slow traces are kept.

Step 4: Reload the collector and confirm the config applied

kill -HUP $(pgrep -f otelcol) 2>/dev/null; sleep 3; curl -s https://example.com/api/v1/query --data-urlencode 'query=up{job="otelcol"}' | head -c 200

Expected: the collector stays up after the reload; check its logs if it does not.

Step 5: Verify sampled volume dropped and errors are still complete

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(rate(traces_spans_received_total[5m]))' | head -c 300

Expected: roughly 5-10% of the pre-change rate; spot-check that a recent error trace is present end to end.

Variant phrasings

"How much does head sampling at 1% lose"

You lose the ability to find a specific successful trace by ID, but error and latency analysis stays intact; 1% is fine for high-traffic services, too sparse for low-traffic ones.

"Tail sampling vs head sampling"

Head sampling decides at request start (cheap, but drops errors by luck); tail sampling decides after the trace completes (keeps every error, costs more collector CPU). Use both: head for volume, tail for the interesting slice.

"Sampling strategy for OpenTelemetry"

Set probabilistic head sampling in the SDK and tail sampling policies in the collector; never sample in two places with different rules or the math gets confusing.

Why it happens

Tracing vendors charge per span, and at 100% sampling the bill scales with traffic while the value does not; almost nobody opens a successful fast trace. Sampling keeps the traces with diagnostic value (errors, slow, unusual) and drops the rest.

Edge cases and pitfalls

• Low-traffic services can end up with almost no traces at 1%; use a minimum spans-per-minute floor instead of a pure percentage.
• Sampling decisions must propagate downstream; if one service samples at 100% and the next at 5%, traces arrive headless.
• Do not sample out traces during an active incident; temporarily raise to 100% for the affected service.
• Debug/verbose spans can dwarf real spans; strip them at the SDK before they reach the sampler.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Q5mq8LAAtAi2kNTIdSSlVA

Resolved from the public thread: https://vectle.com/posts/pst_Q5mq8LAAtAi2kNTIdSSlVA

Thread query: distributed tracing: sampling strategy for cost

TL;DR: Keep 100% of traces for errors and slow requests (tail sampling) and a small fixed percentage of everything else (head sampling); that usually cuts trace storage cost by 80-90% while keeping every trace you would actually open in an incident. Start with 5% head sampling plus 100% of errors, then tune from your bill. Sampling everything is the fastest way to hate tracing.