Diag: agent OOM-killed, find the cardinality or tailing driving memory

TL;DR

For "dmesg | grep -i oom": symptom (exact): agent process dies, dmesg shows the OOM killer took it, supervisor restarts it, repeat. Correlate the start of growth with a deploy or config change.

dmesg | grep -i oom

Steps

1. Symptom (exact): agent process dies, dmesg shows the OOM killer took it, supervisor restarts it, repeat. Host flaps in Infrastructure.
2. Likely causes: (a) a check with exploding cardinality (thousands of instances), (b) heavy log tailing with big buffers, (c) APM trace agent queuing under a trace flood, (d) tiny box, agent sized for bigger.
3. dmesg | grep -i oom: confirms the killer and the victim process.
4. Agent memory over time: does it climb steadily (leak-ish growth, usually cardinality) or spike (burst of checks/logs)?
5. Which check has the most instances? agent status check sections list instance counts. Thousands of instances of one check is the smoking gun.
6. Correlate the start of growth with a deploy or config change.
7. - Cap the offending check: limit instances, increase the check interval, drop high-cardinality tags.

• Trim log tailing: fewer files, higher min log level, or ship the firehose elsewhere.
• Give the box more RAM if the workload is legitimately big; set container memory limits with headroom so the killer takes something expendable.
• Update the agent: memory bugs get fixed, and old versions have known leaks.

5. Verify: memory flat across two full check cycles and a traffic peak, no OOM lines in dmesg for 24h. Alert on agent memory so the next growth pages before the killer does.

When to use

You are seeing this: Host flaps in Infrastructure. Use this skill when you run into "Diag: agent OOM-killed, find the cardinality or tailing driving memory".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

The agent collects 75-100 system metrics per cycle plus check and log workloads; high-cardinality checks and heavy log tailing are the documented memory drivers. Verified against live docs 2026-09-26.