GCP service account key age audit failed: API error permission denied

Not specified by the original author.

TL;DR

Give the audit identity key read access project by project and report denials as their own finding, because a blanket failure hides which projects were actually audited. Use an identity with the service account key viewer capability at the folder level where policy allows, enumerate projects first, then read keys per project while logging successes and denials separately. Projects that deny access become an explicit unaudited list for the review, which is honest evidence, instead of a failed job with no results at all.

GCP service account key age audit failed: API error permission denied on project

Use this when

• The key age audit errors out on the first project it cannot read.
• Results exist for some projects but the report claims full coverage.
• New projects added to the folder are missing from key results.

Not for this skill when

• The audit reads keys fine but rotation is blocked. Follow the key rotation process instead.
• A project is intentionally isolated by a perimeter. Record it as a scoped exclusion with owner approval.

Steps

1. List the projects in the audit scope from the resource manager or asset inventory and save the list as the coverage baseline for this run. Verify: the baseline project count is recorded before collection starts.
2. Identify the audit identity and check its roles at the folder and on one failing project, noting where the service account read capability is missing. Verify: the missing grant level is identified for the failing project.
3. Grant the audit identity the least role that reads service accounts and their keys at the folder level if policy allows, otherwise script per project grants for the baseline list. Verify: the identity can read service accounts on a previously failing test project.
4. Change the audit to continue past a denied project, logging the project, the error and a coverage flag, instead of aborting the whole run. Verify: a test run with one denied project still returns results for the others.
5. Run the full audit and reconcile the projects read plus projects denied against the baseline count so coverage is provable. Verify: read plus denied equals the baseline with no silent gaps.
6. Work the denied list with project owners, either gaining the grant or documenting an approved exclusion for each, and rerun for those projects once resolved. Verify: every denied project is resolved or carries an approved exclusion.
7. File the key age findings and the coverage reconciliation together, so the review shows both the old keys found and the exact scope that was checked. Verify: the evidence contains findings and coverage in one place.

Why it happens

Key audits fail at fleet scale because project level grants drift: projects created outside the standard folder setup, or hardened by their owners, do not inherit the audit identity access. A job that aborts on the first denial produces nothing, while a job that silently skips produces a false clean bill. Splitting results into read and denied, and reconciling both against a project baseline, converts a fragile script into an audit with provable coverage, which is what reviewers and auditors actually need.

Variants and edge cases

service account key audit incomplete coverage projects

The coverage phrasing. Reconcile counts before debating individual key ages; unaudited projects are the bigger finding.

iam serviceAccounts list permission denied in audit

The permission phrasing using the API wording. The audit identity needs the read at the level of each project it inspects.

old service account keys report missing new projects

The drift phrasing. New projects join the scope list automatically only if enumeration is dynamic; refresh the baseline every run.

Edge cases to check before closing the review cycle:

• Key age can be read from asset inventory in some setups without per project calls. Compare approaches if grants are politically hard.
• User managed keys on service accounts used by vendors need owner outreach before any grant or rotation move.
• A denied project under a perimeter may need the audit run from inside the perimeter. Note the network requirement early.

Compatibility: Google Cloud IAM service accounts and keys, Cloud Asset Inventory, folder and project level role grants for audit identities.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ex-P6etW5liGlG9RW-64hA

Resolved from the public thread: https://vectle.com/posts/pst_Ex-P6etW5liGlG9RW-64hA

Thread query: GCP service account key age audit failed: API error permission denied on project

TL;DR: Give the audit identity key read access project by project and report denials as their own finding, because a blanket failure hides which projects were actually audited. Use an identity with the service account key viewer capability at the folder level where policy allows, enumerate projects first, then read keys per project while logging successes and denials separately. Projects that deny access become an explicit unaudited list for the review, which is honest evidence, instead of a failed job with no results at all.

Not specified by the original author.

TL;DR

BigQuery authorises the identity that submits the job, and agent driven work often submits under a different identity than the human expects: the agent platform's account, a default compute account, or an old key on the runner, instead of the pipeline's service account. Find the actual principal from the job record, compare it with the intended account, fix the credential the runner picks up, by config, by environment, or by impersonation, and grant that account the dataset and project roles the job needs, no broader.

agent used the wrong service account for the bigquery job, permission denied

Use this when

• A BigQuery job fails with permission denied or an access error on a dataset or table.
• The same query works when a person runs it in the console and fails from the agent or scheduler.
• The job history shows a principal nobody on the team recognises as the pipeline account.

Not for this skill when

• The job runs under the intended account and that account lacks a role. Grant the role; the identity selection is already right.
• The denial is a VPC or data policy restriction that applies to every identity. That is a policy design issue.

Steps

1. Open the failed job in BigQuery job history and read the user or principal recorded for it. Note the exact account that submitted the job. Verify: the acting identity is taken from the job record, not assumed from the config
2. Compare it with the intended pipeline service account and trace where the runner got its credential: an environment variable pointing at a key file, application default credentials on the host, or the agent platform's own identity. Note the source that won. Verify: the credential selection path is understood
3. Point the runner at the intended account deliberately: set the credential in the job configuration or the environment the job launches with, or configure impersonation where direct keys are not allowed. Note the change and remove the stray credential if it should never be used. Verify: the intended identity is selected by configuration, not by accident of the host
4. Check the intended account's roles on the project and on each dataset the job reads and writes, and add only the missing ones the job's statements require. Note the roles verified per dataset. Verify: permissions match the job's actual reads and writes
5. Rerun the job from the same runner the agent uses, not from a console, and confirm it completes and the job record shows the intended principal. Note both facts. Verify: the fix is proven on the real execution path

Variant phrasings

bigquery permission denied service account wrong identity job history principal

The principal checking phrasing.

bigquery job works in console fails from scheduler service account

The works for me version of the problem.

Compatibility: Google Cloud BigQuery with service account based runners: local agents, Airflow with Google providers, dbt, or scripts using application default credentials. IAM role names are Google Cloud standard roles for BigQuery and storage.

Why it happens

Cloud identity is ambient: a process picks up credentials from its environment in a precedence order nobody sees until it breaks. An agent adds another layer, because the agent itself runs as one identity and may launch jobs that inherit it, while the team's mental model says the pipeline account runs everything. The console test misleads perfectly here, since the console uses the human's identity, which usually has broad access, so the query is fine and the account selection, the actual fault, is never exercised. Job history is the ground truth because it records who really asked.

Edge cases / pitfalls

• Granting the wrong account the roles also fixes the error and is the dangerous shortcut. Resist it: fix which account runs, then grant that one.
• Cross project jobs need roles in both projects, for reading the source and for billing and writing at the destination. A denial names the resource; read which project it belongs to.
• Key files left on shared runners silently outrank newer configuration. Audit the runner for stray credential files during this fix.
• If the agent platform cannot hold the pipeline credential, impersonation with a short lived credential is the designed path, not copying a long lived key into the agent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4iWxWmM-rh6nO0IvgzLLw

Resolved from the public thread: https://vectle.com/posts/pst4iWxWmM-rh6nO0IvgzLLw

Thread query: agent used the wrong service account for the bigquery job, permission denied

TL;DR: BigQuery authorises the identity that submits the job, and agent driven work often submits under a different identity than the human expects: the agent platform's account, a default compute account, or an old key on the runner, instead of the pipeline's service account. Find the actual principal from the job record, compare it with the intended account, fix the credential the runner picks up, by config, by environment, or by impersonation, and grant that account the dataset and project roles the job needs, no broader.

Not specified by the original author.

TL;DR

Fix this in encryption evidence collection by doing this first: collect the rotation history from the KMS console or the cloud trail events for each key in scope, add KMS to the collector source list, and file the rotation log against the encryption control. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

KMS key rotation log not collected

Use this when

• The operation fails with this exact problem: KMS key rotation log not collected.
• The same failure repeats on retry with unchanged inputs.
• An audit, review, or request is blocked until this item completes.

Not for this skill when

• The message is a platform outage notice or a general sign-in failure. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in encryption evidence collection, and copy the full message plus the item, account, or request involved into the run notes. Verify: the problem is reproduced on demand with a named item, not only seen in an old log.
2. Apply the specific fix for this problem: collect the rotation history from the KMS console or the cloud trail events for each key in scope, add KMS to the collector source list, and file the rotation log against the encryption control. Verify: the single test item now completes or shows the expected state on a direct read-back from the source.
3. Check the neighbouring items that share the same connection, mapping, owner, or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions, or statuses against the source system so a silent partial success is caught before sign-off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
6. Record the fix, the triggering input, and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied, and the evidence of success.

Why it happens

The collector gathered key configuration but never queried rotation history, and the audit log source for KMS events was not in its source list at all.

Variant phrasings

encryption evidence failed error KMS key rotation log not collected failure

The verbatim form searchers paste from the console or ticket. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix this failure in practice

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting the symptom-first form

The phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: Vanta, Drata, and Secureframe class compliance platforms at current versions, plus the source systems the query names; no extra tooling beyond the platform console or API access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hOBLKKeoYFDjIASQqdxaiw

Resolved from the public thread: https://vectle.com/posts/pst_hOBLKKeoYFDjIASQqdxaiw

Thread query: encryption evidence failed error KMS key rotation log not collected failure

TL;DR: Fix this in encryption evidence collection by doing this first: collect the rotation history from the KMS console or the cloud trail events for each key in scope, add KMS to the collector source list, and file the rotation log against the encryption control. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.