Diag: agent OOM-killed, find the cardinality or tailing driving memory

Diag: agent OOM-killed

Symptom (exact): agent process dies, dmesg shows the OOM killer took it, supervisor restarts it, repeat. Host flaps in Infrastructure.

Likely causes: (a) a check with exploding cardinality (thousands of instances), (b) heavy log tailing with big buffers, (c) APM trace agent queuing under a trace flood, (d) tiny box, agent sized for bigger.

Confirm:

1. dmesg | grep -i oom: confirms the killer and the victim process.
2. Agent memory over time: does it climb steadily (leak-ish growth, usually cardinality) or spike (burst of checks/logs)?
3. Which check has the most instances? agent status check sections list instance counts. Thousands of instances of one check is the smoking gun.
4. Correlate the start of growth with a deploy or config change.

Fix:

• Cap the offending check: limit instances, increase the check interval, drop high-cardinality tags.
• Trim log tailing: fewer files, higher min log level, or ship the firehose elsewhere.
• Give the box more RAM if the workload is legitimately big; set container memory limits with headroom so the killer takes something expendable.
• Update the agent: memory bugs get fixed, and old versions have known leaks.

Verify: memory flat across two full check cycles and a traffic peak, no OOM lines in dmesg for 24h. Alert on agent memory so the next growth pages before the killer does.

The agent collects 75-100 system metrics per cycle plus check and log workloads; high-cardinality checks and heavy log tailing are the documented memory drivers. Verified against live docs 2026-09-26.