Immutable solution version
DogStatsD dropping metrics under load: socket buffer and sampling
Complete solution
DogStatsD runs over UDP, so a saturated socket buffer drops packets silently. Spotty custom metrics under load mean the buffer, not Datadog.
DogStatsD metrics dropping under load
What you see: custom metrics are fine at low traffic and spotty or gappy at peak. No errors anywhere, because UDP does not do errors.
Why
DogStatsD is UDP. When the app emits faster than the agent drains the socket buffer, the kernel drops packets silently. Common amplifiers:
- Bursty emission (10k increments in a loop, then nothing).
- Many containers sharing one agent with small buffers.
- Histograms with high cardinality: more series, more packets.
Diagnose
The agent exposes its own DogStatsD telemetry: packets received vs dropped. If drops climb with load, it is the buffer. Also check the app side: emitting synchronously in the request path makes the app pay for every packet.
Fix
- Raise the socket buffer (
SO_RCVBUF) on the agent host if the OS default is small. This is the usual fix. - Sample client-side. DogStatsD supports sample rates; a 0.1 rate on a high-volume counter keeps the shape and drops 90 percent of packets.
- Aggregate more, emit less. Counters and distributions exist so you do not emit a gauge per event.
- Prefer distributions over histograms for latency: server-side aggregation handles the math.
- If loss is unacceptable for a specific metric, send it via the API instead of UDP. Slower, reliable, and rate-limited, pick your tradeoff.
Trap
Assuming gaps mean Datadog is down. Check the agent-local drops first; it is almost always the buffer.
Evidence
The DogStatsD docs describe UDP port 8125 submission with 10 second server-side aggregation; UDP delivery is inherently lossy and the agent exposes DogStatsD telemetry for diagnosing drops. Verified against live docs 2026-09-26.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_T4LK6UQVNrtcMSuI3oun9g" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">DogStatsD dropping metrics under load: socket buffer and sampling <span>— Vectle</span></a>