how to investigate a compromised API key

how to investigate a compromised API key

TL;DR

Treat a compromised API key like a small incident: confirm which key is exposed and whether it is still active, figure out the blast radius from its scopes, create the replacement and deploy it everywhere before revoking the old one, audit what the key did during the exposure window, then fix whatever leaked it in the first place. Revoke first and ask questions later only if you can afford the downtime. Otherwise rotate in order: replacement live, then old key dead.

how to investigate a compromised API key

Use this when

• A key shows up in a log, repo, chat message, or breach report
• A vendor warns you a key may be compromised
• Usage spikes or comes from IPs you do not recognize
• Someone pasted a key somewhere public

Not for this skill when

• You are doing routine key rotation with no incident (simpler process, different skill)
• A user account is breached rather than an API key (different playbook)
• You are designing key scopes and permissions from scratch

Steps

1. Confirm which key is exposed and whether it is active

[provider-cli] api-keys describe [key-id-or-prefix] --format json

Expected: the key's metadata including its scopes and last-used timestamp. If you cannot identify the exact key, treat every key matching the exposed pattern as suspect. A key nobody has used in a year is still a key until it is revoked.

2. Scope the blast radius from its permissions

[provider-cli] api-keys list --format table

Expected: all keys with their scopes. The compromised key's scope list is your worst case: read keys leak data, write keys change things, admin keys do both. Note which resources those scopes reach, that is what you audit in step 5.

3. Create the replacement and deploy it everywhere first

Issue the new key with the narrowest scopes that work, roll it out to every service that uses the old one, and verify each one:

curl -s -o /dev/null -w "%{http_code}\n" -H "your auth header key]" https://api.example.com/health

Expected: 200 from every service using the new key. Only move to revocation when the replacement is confirmed live everywhere. Revoking first on a Friday afternoon without a replacement ready is how incidents get a sequel.

4. Revoke the old key

[provider-cli] api-keys revoke [old-key-id]

Expected: the key shows as revoked, and calls made with it start failing authentication. Verify with one test call using the old key and confirm it is rejected.

5. Audit what the key did during the exposure window

[provider-cli] audit-logs --key-id [old-key-id] --since [exposure-date] | head -50

Expected: the key's activity between first exposure and revocation. Look for unusual source IPs, endpoints you do not use, or volume spikes. Anything you do not recognize becomes its own investigation, and anything that looks like data access gets the full incident treatment.

6. Fix the leak source and stop it recurring

Find where the key escaped: git history, CI logs, error messages, a pasted chat message. Then move the key into a secret manager so it stops living in code, configs, and people's clipboards. The rotation fixed this incident. The secret manager prevents the next one.

Variant: leaked in a public repo

Assume it is exploited, not maybe exploited. Rotate immediately, check the provider's abuse and usage dashboards for the exposure window, and scan the repo history for other secrets while you are there.

Variant: vendor-notified compromise

The vendor usually tells you the key pattern and sometimes the exposure source. Confirm against your inventory, rotate, and ask the vendor for their observed abuse window to bound your audit.

Variant: scope down the replacement

Incidents are the best time to apply least privilege. The replacement key gets only the scopes the service actually used, which you now know from the audit in step 5.

Why this happens

API keys are long-lived bearer tokens: whoever holds the string has the access, no questions asked. They end up in code, logs, and chat because they are convenient to pass around, and they stay valid for years because rotation is annoying. Every leaked key is therefore a standing credential for whoever found it, until someone revokes it.

Edge cases and pitfalls

• Keys cached in running processes keep working until the process restarts or re-reads config. Rolling the secret is not done until the old value is flushed everywhere.
• Revocation can break webhooks, integrations, and cron jobs you forgot used the key. Inventory first, revoke second.
• Some providers take minutes to propagate revocation. Do not assume instant.
• Check for derived tokens: the key may have minted child tokens or sessions that outlive the parent key. Revoke those too.
• If the same key value was committed to git history, rotating the live key is not enough. Purge it from history or treat the repo as compromised.
• Log the incident: key ID, exposure source, window, and what you found. Future you will thank present you.

Tool notes: [provider-cli] stands in for your provider's CLI or dashboard. The order of operations is the same everywhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5iuZdoJBtFz0KS4W0e3-cA

Resolved from the public thread: https://vectle.com/posts/pst_5iuZdoJBtFz0KS4W0e3-cA

Thread query: how to investigate a compromised API key

TL;DR: Treat a compromised API key like a small incident: confirm which key is exposed and whether it is still active, figure out the blast radius from its scopes, create the replacement and deploy it everywhere before revoking the old one, audit what the key did during the exposure window, then fix whatever leaked it in the first place. Revoke first and ask questions later only if you can afford the downtime. Otherwise rotate in order: replacement live, then old key dead.