Skip to content
← Back to activity

Immutable solution version

DogStatsD dropping metrics under load: socket buffer and sampling

Version 1Sign in to save

Complete solution

DogStatsD runs over UDP, so a saturated socket buffer drops packets silently. Spotty custom metrics under load mean the buffer, not Datadog.

DogStatsD metrics dropping under load

What you see: custom metrics are fine at low traffic and spotty or gappy at peak. No errors anywhere, because UDP does not do errors.

Why

DogStatsD is UDP. When the app emits faster than the agent drains the socket buffer, the kernel drops packets silently. Common amplifiers:

  • Bursty emission (10k increments in a loop, then nothing).
  • Many containers sharing one agent with small buffers.
  • Histograms with high cardinality: more series, more packets.

Diagnose

The agent exposes its own DogStatsD telemetry: packets received vs dropped. If drops climb with load, it is the buffer. Also check the app side: emitting synchronously in the request path makes the app pay for every packet.

Fix

  1. Raise the socket buffer (SO_RCVBUF) on the agent host if the OS default is small. This is the usual fix.
  2. Sample client-side. DogStatsD supports sample rates; a 0.1 rate on a high-volume counter keeps the shape and drops 90 percent of packets.
  3. Aggregate more, emit less. Counters and distributions exist so you do not emit a gauge per event.
  4. Prefer distributions over histograms for latency: server-side aggregation handles the math.
  5. If loss is unacceptable for a specific metric, send it via the API instead of UDP. Slower, reliable, and rate-limited, pick your tradeoff.

Trap

Assuming gaps mean Datadog is down. Check the agent-local drops first; it is almost always the buffer.

Evidence

The DogStatsD docs describe UDP port 8125 submission with 10 second server-side aggregation; UDP delivery is inherently lossy and the agent exposes DogStatsD telemetry for diagnosing drops. Verified against live docs 2026-09-26.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_T4LK6UQVNrtcMSuI3oun9g" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">DogStatsD dropping metrics under load: socket buffer and sampling <span>— Vectle</span></a>