Diag: agent wont start over the API key, find where it should live

TL;DR

Likely causes: (a) key never configured, (b) configured in the wrong place for the install type, (c) env var lost on restart (set in shell, not in the service definition).

Steps

1. Symptom (exact): agent will not start, log says the API key is missing or invalid (variants: key not set, key empty, authentication failed on first flush).
2. Read the first 50 lines of the agent log after a start attempt. The error names the problem: missing key vs rejected key.
3. Host install: grep -i api_key /etc/datadog-agent/datadog.yaml (the key should be present, value redacted in your head, never pasted anywhere).
4. Container install: kubectl describe / docker inspect the agent and read back DD_API_KEY. Empty means the secret reference broke.
5. If the key is present but rejected, it is cause (c) from the 403 runbook: validate the key in the app.
6. - Host: put the org API key in datadog.yaml and restart.

• Container: fix the secret/env reference and redeploy the agent. Setting it with export in an SSH session does nothing for the systemd/docker service.
• Never bake the key into an image. Secret manager or orchestrator secret, referenced at runtime.

4. Verify: datadog-agent status runs and shows a key fingerprint line without auth errors; forwarder accepts payloads within a minute.

When to use

You are seeing this: Symptom (exact): agent will not start, log says the API key is missing or invalid (variants: key not set, key empty, authentication failed on first flush). Use this skill when you run into "Diag: agent wont start over the API key, find where it should live".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

The agent getting-started docs require the API key and site as the two mandatory config values; container installs take them as DDAPIKEY / DD_SITE env vars while hosts use datadog.yaml. Verified against live docs 2026-09-26.