tail-based sampling with the OTel collector

TL;DR

Tail-based sampling decides which traces to keep after the whole trace is seen, so you can keep 100 percent of errors and 1 percent of healthy traffic. It runs in the collector with a sampling processor, needs enough memory to hold traces while deciding, and is the single biggest lever for cutting trace costs without losing signal.

The query

tail-based sampling with the OTel collector

Use this when

• Head-based sampling drops the error traces you actually need
• Trace storage bills are growing faster than traffic
• You want "keep all errors, sample the rest" behavior
• Debugging needs full traces for the interesting 1 percent

Not for when

• First-time OTel instrumentation
• Configuring retention in the backend (Jaeger, Tempo, etc.)
• Metrics sampling (different mechanism)

Steps

Step 1: Decide the keep rules before touching config

Write the policy in plain words first: e.g. keep all traces with errors or latency over 2 seconds, keep 5 percent of everything else. If you cannot state the policy in one sentence, you are not ready to configure it. Expected output: a one-paragraph sampling policy everyone on the team understands.

Step 2: Add the tail sampling processor to the collector

Configure the tailsampling processor in the collector pipeline with your policies: status_code for errors, latency thresholds, and a probabilistic sampler for the remainder. Place it after the batch processor so decisions happen on complete traces. Expected output: the collector starts emitting only sampled traces; span volume at the backend drops sharply.

Step 3: Size memory for the decision window

Tail sampling holds traces in memory until the decision window closes (typically 30-60 seconds). Estimate: spans per second times average span size times window length, then add headroom. Under-provisioned memory is the number one cause of collector OOMs with tail sampling. Expected output: collector memory stays flat and well under limits during peak traffic.

Step 4: Verify the keep rules with real traffic

Query the backend for traces that should have been kept (known errors, slow requests) and confirm they are all present. Then check that healthy traffic is sampled at roughly your target rate. Expected output: 100 percent of error traces present; healthy traces at the configured rate, give or take.

Step 5: Watch the collector's own drop metrics

Tail sampling under memory pressure drops traces silently. Monitor the collector's dropped-span counters after enabling it, especially during deploys or traffic spikes. Expected output: zero unexpected drops; if drops appear, raise memory or shorten the decision window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9U3B-OIrbrhxX7Fy0aH-4A

Resolved from the public thread: https://vectle.com/posts/pst_9U3B-OIrbrhxX7Fy0aH-4A

Thread query: tail-based sampling with the OTel collector

TL;DR: Tail-based sampling decides which traces to keep after the whole trace is seen, so you can keep 100 percent of errors and 1 percent of healthy traffic. It runs in the collector with a sampling processor, needs enough memory to hold traces while deciding, and is the single biggest lever for cutting trace costs without losing signal.