DogStatsD dropping metrics under load: socket buffer and sampling

TL;DR: DogStatsD runs over UDP, so a saturated socket buffer drops packets silently. Spotty custom metrics under load mean the buffer, not Datadog. Aggregate more, emit less. Prefer distributions over histograms for latency: server-side aggregation handles the math.

The fix

1. Bursty emission (10k increments in a loop, then nothing).
2. Many containers sharing one agent with small buffers.
3. Histograms with high cardinality: more series, more packets.
4. Raise the socket buffer (SO_RCVBUF) on the agent host if the OS default is small. This is the usual fix.
5. Sample client-side. DogStatsD supports sample rates; a 0.1 rate on a high-volume counter keeps the shape and drops 90 percent of packets.
6. Aggregate more, emit less. Counters and distributions exist so you do not emit a gauge per event.
7. Prefer distributions over histograms for latency: server-side aggregation handles the math.
8. If loss is unacceptable for a specific metric, send it via the API instead of UDP. Slower, reliable, and rate-limited, pick your tradeoff.

When to use this

• This covers exactly what the title says: DogStatsD dropping metrics under load.
• You are setting this up for the first time, or auditing an existing setup.
• You want the key gotchas in one place before you start.

When not to use this

• You are doing a different workflow with DogStatsD; these steps are specific to the title above.
• You need the full reference docs; this is the short path, not the manual.

Compatibility

• Mentions 0.1 in the original; behavior may differ on much older or newer releases.