exposed gcp service account key: rotation failed

Not specified by the original author.

TL;DR

Never delete an exposed service account key before its replacement is live everywhere; do create, roll out, then delete. Create the new key, update every consumer, verify, and only then delete the old key. Deleting first causes outages; deleting never leaves the exposure open.

Error

exposed gcp service account key: rotation failed

Steps

1. Create the replacement: run gcloud iam service-accounts keys create for the service account, saving the new key securely. Expected: a new key ID alongside the old one.
2. Inventory consumers: search CI configs, servers, and apps for the old key's usage (key ID or file). Expected: a complete list of everything that must move.
3. Roll out the new key to every consumer, one environment at a time, verifying each. Expected: all consumers authenticate with the new key.
4. Verify no traffic uses the old key: check audit logs for authentications with the old key ID. Expected: zero recent uses before you delete.
5. Delete the old key: run gcloud iam service-accounts keys delete with the old key ID. Expected: the old key is gone and cannot authenticate.

When to use

• A GCP service account key was exposed and must be rotated
• Planned key rotation for hygiene
• Migrating consumers off a shared key

When not to use

• Deciding to move to workload identity (migration project, then delete keys)
• Key already deleted but consumers still reference it (restore rollout order)

Tool compatibility

• Google Cloud IAM service account keys (gcloud iam service-accounts keys); Cloud Audit Logs

Variant phrasings

gcp key rotation caused outage

The old key was deleted before rollout finished; recreate the rollout order and keep both keys live during migration.

find where service account key is used

Audit logs keyed by key ID show recent usage; combine with a config search for full coverage.

Why it happens

Service account keys are long-lived bearer credentials copied to many places. Rotation fails when teams delete first (outage) or forget a consumer (the old key stays valid somewhere), so the create-rollout-verify-delete sequence is the only safe order.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5qosguxECD9cqZY3nO-xVg

Resolved from the public thread: https://vectle.com/posts/pst_5qosguxECD9cqZY3nO-xVg

Thread query: exposed gcp service account key: rotation failed

TL;DR: Never delete an exposed service account key before its replacement is live everywhere; do create, roll out, then delete. Create the new key, update every consumer, verify, and only then delete the old key. Deleting first causes outages; deleting never leaves the exposure open.