Diag: empty metrics explorer, confirm submission before blaming the query

TL;DR

Symptom (exact): agent installed and running, but Metrics Explorer shows no data for the host, not even system metrics. If 403/connection errors, stop here and run the 403 or proxy runbook.

Steps

1. Likely causes: (a) agent not submitting (key/site), (b) querying the wrong thing, (c) clock skew.
2. datadog-agent status: forwarder accepting payloads? If 403/connection errors, stop here and run the 403 or proxy runbook. This is not a metrics problem.
3. In the app, Infrastructure, Hosts: is the host listed at all? Not listed means nothing is submitted (cause a). Listed but metrics empty is weirder, keep going.
4. Query system.cpu.user with no tags and a 1-hour window. System metrics are unconditional: if even these are missing, submission is broken.
5. Check the host clock against NTP. Skew breaks TLS to intake.
6. Fix: fix submission (key/site/proxy), or fix the query (metric name spelling, tag filters, wider window).
7. Verify: system.cpu.user graphs within two minutes of the fix. Then check your custom metric. Rule: system metrics first, custom second. If system metrics flow and yours does not, the emitter is the problem, not the agent.

When to use

You are seeing this: Likely causes: (a) agent not submitting (key/site), (b) querying the wrong thing, (c) clock skew. Use this skill when you run into "Diag: empty metrics explorer, confirm submission before blaming the query".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

The agent getting-started docs: the agent collects 75-100 system metrics every 15-20 seconds with datadog.agent.up OK on connect; the site/key docs cover the ghost-agent failure modes. Verified against live docs 2026-09-26.