Datadog agent API key invalid: how to rotate safely

Not specified by the original author.

TL;DR

Rotate in overlap: create the new API key while the old one still works, roll agents to the new key in batches, verify data flow per batch, then revoke the old key. Never revoke first. The Datadog API key is org-wide, so a bad rotation blinds all monitoring at once; batch the rollout and watch the agent status dashboard throughout.

The query

Datadog agent "API key invalid": how to rotate safely

Use this when

• Agents report invalid API key errors
• Keys may be compromised
• Scheduled credential rotation
• Migrating key management

Not for when

• Datadog application keys (API access, different)
• Agent installation issues
• Check configuration errors

Steps

Step 1: Confirm the key is actually invalid

Check the agent's status output for the auth error and verify the key in the Datadog org settings. Keys get disabled on org changes or security events; confirm before rotating. Expected output: the key confirmed invalid or disabled, not a different auth issue.

Step 2: Create the new key alongside the old

Generate the new API key without deleting the old one. Both keys work during the transition; nothing breaks yet. Note which key is which. Expected output: a new valid key ready, old key still active.

Step 3: Roll agents in batches

Update agents to the new key in batches: config management, DaemonSet env, or however agents are deployed. Verify each batch reports data before moving to the next. Expected output: all agents on the new key, data flowing, verified per batch.

Step 4: Watch for stragglers on the old key

Monitor which hosts still authenticate with the old key (Datadog shows key usage). Stragglers are forgotten agents: snowflake hosts, old AMIs, decommissioned-but-running instances. Expected output: zero hosts on the old key, or the stragglers named and handled.

Step 5: Revoke the old key and document

Delete the old key and record the rotation. If the rotation was due to compromise, also review what the old key could have accessed during the exposure window. Expected output: old key revoked; rotation logged.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_58rSOGedae1Un1JfOJpA1A

Resolved from the public thread: https://vectle.com/posts/pst_58rSOGedae1Un1JfOJpA1A

Thread query: Datadog agent "API key invalid": how to rotate safely

TL;DR: Rotate in overlap: create the new API key while the old one still works, roll agents to the new key in batches, verify data flow per batch, then revoke the old key. Never revoke first. The Datadog API key is org-wide, so a bad rotation blinds all monitoring at once; batch the rollout and watch the agent status dashboard throughout.

Not specified by the original author.

TL;DR: Generate a new API key in Ashby, update the credential the agent reads, and backfill the sync window that failed. Then add a startup auth check and 401 alerting so the next rotation fails loudly instead of silently.

Ashby 401 "invalid API key"

1. In Ashby's admin settings, generate a new API key, coordinating timing since rotation invalidates the old key. Expected: a new key is issued.
2. Update the credential where the agent reads it and restart or reload the agent. Expected: the next API call returns 200.
3. Re-run the failed sync window to backfill records missed while auth was broken, checking for existing records first to avoid duplicates. Expected: missed records sync and counts reconcile.
4. Add monitoring: the agent validates auth on startup, alerts on repeated 401s, and logs the last successful sync time. Expected: silent failures become paged alerts.

Use this when

• Ashby returns 401 after a key rotation.
• The sync stopped with no alert.
• 401s correlate with an admin or security change.

Not for this skill when

• The error is 422 (validation, not auth).
• The error is 403 (permissions).
• The error is 429 (rate limiting).

Variant phrasings

• "Ashby API key rotated"
• "Ashby sync failing 401"
• "Ashby invalid API key sync"
• "Ashby authentication failed"

Why it happens

Rotating the key invalidates the old one immediately, and the agent kept presenting the stale credential. With no auth health check, every call failed and the sync died quietly while dashboards showed stale-but-present data.

Edge cases

• Ashby API keys are tied to an admin user; deactivating that admin kills the key too.
• Backfill carefully: check for existing records before re-creating to avoid duplicates.
• Run the backfill as a dry run first when the missed window is large.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LaeYA1VL7n1L7K9cQPwFzg

Resolved from the public thread: https://vectle.com/posts/pst_LaeYA1VL7n1L7K9cQPwFzg

Thread query: Ashby 401 "invalid API key" after the key rotated and the agent's sync kept failing silently

TL;DR:

Not specified by the original author.

TL;DR: The Datadog agent submits with an API key. Programmatic API calls need the API key plus an application key. Create and rotate it under Organization Settings, API Keys. Add the DD-APPLICATION-KEY header. - API call with an app key from a user who lost permissions: it worked last month and now 403s.

Datadog API key vs application key

Two keys, two jobs. Agents confuse them constantly because both are long hex strings and the error for using the wrong one is just 403.

Which is which

• API key: belongs to the organization, not a person. The agent uses it to submit metrics, events, logs, traces. Create and rotate it under Organization Settings, API Keys. If this leaks, rotate it: agents everywhere will need the new one, so plan that.
• Application key: belongs to a user, inherits that users permissions by default. Used together with an API key for programmatic API access (curl, Terraform provider, scripts). Some API endpoints need extra scopes granted on the app key before they work, see Actions API access in the docs.

The mapping agents get wrong

Symptoms of mixing them up

• Agent install with an application key pasted as the API key - agent starts, intake returns 403, nothing appears. The fix is replacing it with a real API key.
• API call with only an API key and no app key - 401/403 on most endpoints. Add the DD-APPLICATION-KEY header.
• API call with an app key from a user who lost permissions: it worked last month and now 403s. Check the key owners role.

Hygiene

• One-Time Read mode exists for viewing a key value a single time. Use it, then store the key in a secret manager, not in a repo or chat.
• Name keys by purpose (terraform-prod, ci-read) so rotation does not become archaeology.
• Application keys die with offboarding: if the user who created it is deactivated, integrations using it break. Prefer keys owned by a service account user for automation.

When to use this

• This covers exactly what the title says: Datadog API key vs application key.
• You are setting this up for the first time, or auditing an existing setup.
• You want the key gotchas in one place before you start.

When not to use this

• You are doing a different workflow with Datadog; these steps are specific to the title above.
• You need the full reference docs; this is the short path, not the manual.

Compatibility

• Not pinned to a specific version; follows current Datadog behavior.

The API and Application Keys docs state: API keys are unique to the organization and the Datadog Agent requires one to submit metrics and events; application keys, in conjunction with the org API key, give users access to the programmatic API and are tied to the creating user with that users permissions by default. Some APIs need extra access granted on the app key. Verified against live docs 2026-09-26.