Diag: agent wont start over the API key, find where it should live

Diag: agent fails at startup on the API key

Symptom (exact): agent will not start, log says the API key is missing or invalid (variants: key not set, key empty, authentication failed on first flush).

Likely causes: (a) key never configured, (b) configured in the wrong place for the install type, (c) env var lost on restart (set in shell, not in the service definition).

Confirm:

1. Read the first 50 lines of the agent log after a start attempt. The error names the problem: missing key vs rejected key.
2. Host install: grep -i api_key /etc/datadog-agent/datadog.yaml (the key should be present, value redacted in your head, never pasted anywhere).
3. Container install: kubectl describe / docker inspect the agent and read back DD_API_KEY. Empty means the secret reference broke.
4. If the key is present but rejected, it is cause (c) from the 403 runbook: validate the key in the app.

Fix:

• Host: put the org API key in datadog.yaml and restart.
• Container: fix the secret/env reference and redeploy the agent. Setting it with export in an SSH session does nothing for the systemd/docker service.
• Never bake the key into an image. Secret manager or orchestrator secret, referenced at runtime.

Verify: datadog-agent status runs and shows a key fingerprint line without auth errors; forwarder accepts payloads within a minute.

The agent getting-started docs require the API key and site as the two mandatory config values; container installs take them as DDAPIKEY / DD_SITE env vars while hosts use datadog.yaml. Verified against live docs 2026-09-26.