how to debug OTel collector pipeline drops

TL;DR

When the OTel collector drops data, the pipeline is almost always the culprit: a processor or exporter that cannot keep up, and a queue that overflows silently. The fix is to find which stage is dropping (receivers, processors, exporters each report it), then give that stage room: bigger queues, more batching, or fewer processors in the path. Most pipeline drops disappear once you look at the collector's own metrics.

The query

how to debug OTel collector pipeline drops

Use this when

• Spans or metrics stop arriving at your backend but the app looks fine
• The collector logs dropped spans or queue-full messages
• Telemetry gaps line up with traffic spikes
• You are running the collector as a gateway or sidecar

Not for when

• Instrumenting an application for the first time (SDK setup)
• Choosing between tracing backends
• Debugging the application's own logic

Steps

Step 1: Check the collector's own telemetry

Scrape the collector's Prometheus metrics endpoint (default port 8888) and look at the pipeline counters: accepted vs refused vs dropped spans for each receiver, processor, and exporter. Expected output: one stage shows dropped counts climbing while the others look healthy. That stage is your problem.

Step 2: Inspect the logs for queue pressure

Search the collector logs for words like "dropped", "queue", and "memory_limiter". The memory limiter processor, if configured, starts refusing data when the collector's memory crosses its threshold, which shows up as drops in the stage right before it. Expected output: a clear log line naming the pipeline or exporter under pressure, e.g. a sending queue reported as full.

Step 3: Tune the exporter's sending queue and retry

In the exporter config, raise sending_queue queuesize and numconsumers, and enable retry_on_failure with sensible max elapsed time. A slow backend makes exporters back up; the queue absorbs bursts so the pipeline does not drop. Expected output: on the next traffic spike, dropped counts stay flat while the queue fills and drains.

Step 4: Simplify the processor chain

Each processor in the pipeline adds latency. If you have several tail-based sampling or batch processors chained, try the pipeline with fewer of them and re-check the drop counters. Batch processors especially should sit just before exporters, with generous timeouts. Expected output: fewer drops and lower collector CPU, with no loss of needed telemetry.

Step 5: Scale the collector horizontally if a single instance is saturated

When one collector instance cannot keep up at peak, run more replicas behind a load balancer (or use the collector's loadbalancing exporter). This is cheaper than tuning one box forever. Expected output: per-instance drop counters fall to zero; total accepted spans match what your SDKs send.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HrT1078tAHN87ExgnxNEbg