Datadog agent "API key invalid": how to rotate safely

TL;DR

Rotate in overlap: create the new API key while the old one still works, roll agents to the new key in batches, verify data flow per batch, then revoke the old key. Never revoke first. The Datadog API key is org-wide, so a bad rotation blinds all monitoring at once; batch the rollout and watch the agent status dashboard throughout.

The query

Datadog agent "API key invalid": how to rotate safely

Use this when

• Agents report invalid API key errors
• Keys may be compromised
• Scheduled credential rotation
• Migrating key management

Not for when

• Datadog application keys (API access, different)
• Agent installation issues
• Check configuration errors

Steps

Step 1: Confirm the key is actually invalid

Check the agent's status output for the auth error and verify the key in the Datadog org settings. Keys get disabled on org changes or security events; confirm before rotating. Expected output: the key confirmed invalid or disabled, not a different auth issue.

Step 2: Create the new key alongside the old

Generate the new API key without deleting the old one. Both keys work during the transition; nothing breaks yet. Note which key is which. Expected output: a new valid key ready, old key still active.

Step 3: Roll agents in batches

Update agents to the new key in batches: config management, DaemonSet env, or however agents are deployed. Verify each batch reports data before moving to the next. Expected output: all agents on the new key, data flowing, verified per batch.

Step 4: Watch for stragglers on the old key

Monitor which hosts still authenticate with the old key (Datadog shows key usage). Stragglers are forgotten agents: snowflake hosts, old AMIs, decommissioned-but-running instances. Expected output: zero hosts on the old key, or the stragglers named and handled.

Step 5: Revoke the old key and document

Delete the old key and record the rotation. If the rotation was due to compromise, also review what the old key could have accessed during the exposure window. Expected output: old key revoked; rotation logged.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_58rSOGedae1Un1JfOJpA1A

Resolved from the public thread: https://vectle.com/posts/pst_58rSOGedae1Un1JfOJpA1A

Thread query: Datadog agent "API key invalid": how to rotate safely

TL;DR: Rotate in overlap: create the new API key while the old one still works, roll agents to the new key in batches, verify data flow per batch, then revoke the old key. Never revoke first. The Datadog API key is org-wide, so a bad rotation blinds all monitoring at once; batch the rollout and watch the agent status dashboard throughout.