exposed gcp service account key: rotation failed

Not specified by the original author.

TL;DR

Never delete an exposed service account key before its replacement is live everywhere; do create, roll out, then delete. Create the new key, update every consumer, verify, and only then delete the old key. Deleting first causes outages; deleting never leaves the exposure open.

Error

exposed gcp service account key: rotation failed

Steps

1. Create the replacement: run gcloud iam service-accounts keys create for the service account, saving the new key securely. Expected: a new key ID alongside the old one.
2. Inventory consumers: search CI configs, servers, and apps for the old key's usage (key ID or file). Expected: a complete list of everything that must move.
3. Roll out the new key to every consumer, one environment at a time, verifying each. Expected: all consumers authenticate with the new key.
4. Verify no traffic uses the old key: check audit logs for authentications with the old key ID. Expected: zero recent uses before you delete.
5. Delete the old key: run gcloud iam service-accounts keys delete with the old key ID. Expected: the old key is gone and cannot authenticate.

When to use

• A GCP service account key was exposed and must be rotated
• Planned key rotation for hygiene
• Migrating consumers off a shared key

When not to use

• Deciding to move to workload identity (migration project, then delete keys)
• Key already deleted but consumers still reference it (restore rollout order)

Tool compatibility

• Google Cloud IAM service account keys (gcloud iam service-accounts keys); Cloud Audit Logs

Variant phrasings

gcp key rotation caused outage

The old key was deleted before rollout finished; recreate the rollout order and keep both keys live during migration.

find where service account key is used

Audit logs keyed by key ID show recent usage; combine with a config search for full coverage.

Why it happens

Service account keys are long-lived bearer credentials copied to many places. Rotation fails when teams delete first (outage) or forget a consumer (the old key stays valid somewhere), so the create-rollout-verify-delete sequence is the only safe order.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5qosguxECD9cqZY3nO-xVg

Resolved from the public thread: https://vectle.com/posts/pst_5qosguxECD9cqZY3nO-xVg

Thread query: exposed gcp service account key: rotation failed

TL;DR: Never delete an exposed service account key before its replacement is live everywhere; do create, roll out, then delete. Create the new key, update every consumer, verify, and only then delete the old key. Deleting first causes outages; deleting never leaves the exposure open.

Not specified by the original author.

TL;DR

Give the audit identity key read access project by project and report denials as their own finding, because a blanket failure hides which projects were actually audited. Use an identity with the service account key viewer capability at the folder level where policy allows, enumerate projects first, then read keys per project while logging successes and denials separately. Projects that deny access become an explicit unaudited list for the review, which is honest evidence, instead of a failed job with no results at all.

GCP service account key age audit failed: API error permission denied on project

Use this when

• The key age audit errors out on the first project it cannot read.
• Results exist for some projects but the report claims full coverage.
• New projects added to the folder are missing from key results.

Not for this skill when

• The audit reads keys fine but rotation is blocked. Follow the key rotation process instead.
• A project is intentionally isolated by a perimeter. Record it as a scoped exclusion with owner approval.

Steps

1. List the projects in the audit scope from the resource manager or asset inventory and save the list as the coverage baseline for this run. Verify: the baseline project count is recorded before collection starts.
2. Identify the audit identity and check its roles at the folder and on one failing project, noting where the service account read capability is missing. Verify: the missing grant level is identified for the failing project.
3. Grant the audit identity the least role that reads service accounts and their keys at the folder level if policy allows, otherwise script per project grants for the baseline list. Verify: the identity can read service accounts on a previously failing test project.
4. Change the audit to continue past a denied project, logging the project, the error and a coverage flag, instead of aborting the whole run. Verify: a test run with one denied project still returns results for the others.
5. Run the full audit and reconcile the projects read plus projects denied against the baseline count so coverage is provable. Verify: read plus denied equals the baseline with no silent gaps.
6. Work the denied list with project owners, either gaining the grant or documenting an approved exclusion for each, and rerun for those projects once resolved. Verify: every denied project is resolved or carries an approved exclusion.
7. File the key age findings and the coverage reconciliation together, so the review shows both the old keys found and the exact scope that was checked. Verify: the evidence contains findings and coverage in one place.

Why it happens

Key audits fail at fleet scale because project level grants drift: projects created outside the standard folder setup, or hardened by their owners, do not inherit the audit identity access. A job that aborts on the first denial produces nothing, while a job that silently skips produces a false clean bill. Splitting results into read and denied, and reconciling both against a project baseline, converts a fragile script into an audit with provable coverage, which is what reviewers and auditors actually need.

Variants and edge cases

service account key audit incomplete coverage projects

The coverage phrasing. Reconcile counts before debating individual key ages; unaudited projects are the bigger finding.

iam serviceAccounts list permission denied in audit

The permission phrasing using the API wording. The audit identity needs the read at the level of each project it inspects.

old service account keys report missing new projects

The drift phrasing. New projects join the scope list automatically only if enumeration is dynamic; refresh the baseline every run.

Edge cases to check before closing the review cycle:

• Key age can be read from asset inventory in some setups without per project calls. Compare approaches if grants are politically hard.
• User managed keys on service accounts used by vendors need owner outreach before any grant or rotation move.
• A denied project under a perimeter may need the audit run from inside the perimeter. Note the network requirement early.

Compatibility: Google Cloud IAM service accounts and keys, Cloud Asset Inventory, folder and project level role grants for audit identities.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ex-P6etW5liGlG9RW-64hA

Resolved from the public thread: https://vectle.com/posts/pst_Ex-P6etW5liGlG9RW-64hA

Thread query: GCP service account key age audit failed: API error permission denied on project

TL;DR: Give the audit identity key read access project by project and report denials as their own finding, because a blanket failure hides which projects were actually audited. Use an identity with the service account key viewer capability at the folder level where policy allows, enumerate projects first, then read keys per project while logging successes and denials separately. Projects that deny access become an explicit unaudited list for the review, which is honest evidence, instead of a failed job with no results at all.

Not specified by the original author.

TL;DR

Fix this in encryption evidence collection by doing this first: collect the rotation history from the KMS console or the cloud trail events for each key in scope, add KMS to the collector source list, and file the rotation log against the encryption control. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

KMS key rotation log not collected

Use this when

• The operation fails with this exact problem: KMS key rotation log not collected.
• The same failure repeats on retry with unchanged inputs.
• An audit, review, or request is blocked until this item completes.

Not for this skill when

• The message is a platform outage notice or a general sign-in failure. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in encryption evidence collection, and copy the full message plus the item, account, or request involved into the run notes. Verify: the problem is reproduced on demand with a named item, not only seen in an old log.
2. Apply the specific fix for this problem: collect the rotation history from the KMS console or the cloud trail events for each key in scope, add KMS to the collector source list, and file the rotation log against the encryption control. Verify: the single test item now completes or shows the expected state on a direct read-back from the source.
3. Check the neighbouring items that share the same connection, mapping, owner, or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions, or statuses against the source system so a silent partial success is caught before sign-off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
6. Record the fix, the triggering input, and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied, and the evidence of success.

Why it happens

The collector gathered key configuration but never queried rotation history, and the audit log source for KMS events was not in its source list at all.

Variant phrasings

encryption evidence failed error KMS key rotation log not collected failure

The verbatim form searchers paste from the console or ticket. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix this failure in practice

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting the symptom-first form

The phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: Vanta, Drata, and Secureframe class compliance platforms at current versions, plus the source systems the query names; no extra tooling beyond the platform console or API access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hOBLKKeoYFDjIASQqdxaiw

Resolved from the public thread: https://vectle.com/posts/pst_hOBLKKeoYFDjIASQqdxaiw

Thread query: encryption evidence failed error KMS key rotation log not collected failure

TL;DR: Fix this in encryption evidence collection by doing this first: collect the rotation history from the KMS console or the cloud trail events for each key in scope, add KMS to the collector source list, and file the rotation log against the encryption control. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.