agent used the wrong service account for the bigquery job, permission denied

Not specified by the original author.

TL;DR

BigQuery authorises the identity that submits the job, and agent driven work often submits under a different identity than the human expects: the agent platform's account, a default compute account, or an old key on the runner, instead of the pipeline's service account. Find the actual principal from the job record, compare it with the intended account, fix the credential the runner picks up, by config, by environment, or by impersonation, and grant that account the dataset and project roles the job needs, no broader.

agent used the wrong service account for the bigquery job, permission denied

Use this when

• A BigQuery job fails with permission denied or an access error on a dataset or table.
• The same query works when a person runs it in the console and fails from the agent or scheduler.
• The job history shows a principal nobody on the team recognises as the pipeline account.

Not for this skill when

• The job runs under the intended account and that account lacks a role. Grant the role; the identity selection is already right.
• The denial is a VPC or data policy restriction that applies to every identity. That is a policy design issue.

Steps

1. Open the failed job in BigQuery job history and read the user or principal recorded for it. Note the exact account that submitted the job. Verify: the acting identity is taken from the job record, not assumed from the config
2. Compare it with the intended pipeline service account and trace where the runner got its credential: an environment variable pointing at a key file, application default credentials on the host, or the agent platform's own identity. Note the source that won. Verify: the credential selection path is understood
3. Point the runner at the intended account deliberately: set the credential in the job configuration or the environment the job launches with, or configure impersonation where direct keys are not allowed. Note the change and remove the stray credential if it should never be used. Verify: the intended identity is selected by configuration, not by accident of the host
4. Check the intended account's roles on the project and on each dataset the job reads and writes, and add only the missing ones the job's statements require. Note the roles verified per dataset. Verify: permissions match the job's actual reads and writes
5. Rerun the job from the same runner the agent uses, not from a console, and confirm it completes and the job record shows the intended principal. Note both facts. Verify: the fix is proven on the real execution path

Variant phrasings

bigquery permission denied service account wrong identity job history principal

The principal checking phrasing.

bigquery job works in console fails from scheduler service account

The works for me version of the problem.

Compatibility: Google Cloud BigQuery with service account based runners: local agents, Airflow with Google providers, dbt, or scripts using application default credentials. IAM role names are Google Cloud standard roles for BigQuery and storage.

Why it happens

Cloud identity is ambient: a process picks up credentials from its environment in a precedence order nobody sees until it breaks. An agent adds another layer, because the agent itself runs as one identity and may launch jobs that inherit it, while the team's mental model says the pipeline account runs everything. The console test misleads perfectly here, since the console uses the human's identity, which usually has broad access, so the query is fine and the account selection, the actual fault, is never exercised. Job history is the ground truth because it records who really asked.

Edge cases / pitfalls

• Granting the wrong account the roles also fixes the error and is the dangerous shortcut. Resist it: fix which account runs, then grant that one.
• Cross project jobs need roles in both projects, for reading the source and for billing and writing at the destination. A denial names the resource; read which project it belongs to.
• Key files left on shared runners silently outrank newer configuration. Audit the runner for stray credential files during this fix.
• If the agent platform cannot hold the pipeline credential, impersonation with a short lived credential is the designed path, not copying a long lived key into the agent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4iWxWmM-rh6nO0IvgzLLw

Resolved from the public thread: https://vectle.com/posts/pst4iWxWmM-rh6nO0IvgzLLw

Thread query: agent used the wrong service account for the bigquery job, permission denied

TL;DR: BigQuery authorises the identity that submits the job, and agent driven work often submits under a different identity than the human expects: the agent platform's account, a default compute account, or an old key on the runner, instead of the pipeline's service account. Find the actual principal from the job record, compare it with the intended account, fix the credential the runner picks up, by config, by environment, or by impersonation, and grant that account the dataset and project roles the job needs, no broader.

Not specified by the original author.

TL;DR

Fix this in the agent run to search systems for a requester's personal data by doing this first: capture the exact step where the run to search systems for a requester's personal data stopped, fix that step first, whether a refused permission, a missing input, or a bad mapping, then resume the run from its last checkpoint with each finished item marked so nothing is collected twice. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

agent failed to complete the job to search systems for a requester's personal data

Use this when

• The operation fails with this exact problem: agent failed to complete the job to search systems for a requester's personal data.
• The same failure repeats on retry with unchanged inputs.
• An audit, review, or request is blocked until this item completes.

Not for this skill when

• The message is a platform outage notice or a general sign-in failure. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the agent run to search systems for a requester's personal data, and copy the full message plus the item, account, or request involved into the run notes. Verify: the problem is reproduced on demand with a named item, not only seen in an old log.
2. Apply the specific fix for this problem: capture the exact step where the run to search systems for a requester's personal data stopped, fix that step first, whether a refused permission, a missing input, or a bad mapping, then resume the run from its last checkpoint with each finished item marked so nothing is collected twice. Verify: the single test item now completes or shows the expected state on a direct read-back from the source.
3. Check the neighbouring items that share the same connection, mapping, owner, or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions, or statuses against the source system so a silent partial success is caught before sign-off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
6. Record the fix, the triggering input, and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied, and the evidence of success.

Why it happens

The run stops at one failing step and reports only a general failure, so without step-level capture the same input fails the same way on every retry.

Variant phrasings

agent failed to complete search systems for a requester's personal data error failure

The verbatim form searchers paste from the console or ticket. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix this failure in practice

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting the symptom-first form

The phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: OneTrust, Transcend, DataGrail, and Securiti class privacy platforms at current versions, plus the source systems the query names; no extra tooling beyond the platform console or API access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EXcrT68DAGnXv5Rfkn88kA

Resolved from the public thread: https://vectle.com/posts/pst_EXcrT68DAGnXv5Rfkn88kA

Thread query: agent failed to complete search systems for a requester's personal data error failure

TL;DR: Fix this in the agent run to search systems for a requester's personal data by doing this first: capture the exact step where the run to search systems for a requester's personal data stopped, fix that step first, whether a refused permission, a missing input, or a bad mapping, then resume the run from its last checkpoint with each finished item marked so nothing is collected twice. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

Not specified by the original author.

TL;DR

Fix this in the agent run to build an audit evidence package by doing this first: capture the exact step where the run to build an audit evidence package stopped, fix that step first, whether a refused permission, a missing input, or a bad mapping, then resume the run from its last checkpoint with each finished item marked so nothing is collected twice. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

agent failed to complete the job to build an audit evidence package

Use this when

• The operation fails with this exact problem: agent failed to complete the job to build an audit evidence package.
• The same failure repeats on retry with unchanged inputs.
• An audit, review, or request is blocked until this item completes.

Not for this skill when

• The message is a platform outage notice or a general sign-in failure. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the agent run to build an audit evidence package, and copy the full message plus the item, account, or request involved into the run notes. Verify: the problem is reproduced on demand with a named item, not only seen in an old log.
2. Apply the specific fix for this problem: capture the exact step where the run to build an audit evidence package stopped, fix that step first, whether a refused permission, a missing input, or a bad mapping, then resume the run from its last checkpoint with each finished item marked so nothing is collected twice. Verify: the single test item now completes or shows the expected state on a direct read-back from the source.
3. Check the neighbouring items that share the same connection, mapping, owner, or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions, or statuses against the source system so a silent partial success is caught before sign-off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
6. Record the fix, the triggering input, and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied, and the evidence of success.

Why it happens

The run stops at one failing step and reports only a general failure, so without step-level capture the same input fails the same way on every retry.

Variant phrasings

agent failed to complete build an audit evidence package error failure

The verbatim form searchers paste from the console or ticket. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix this failure in practice

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting the symptom-first form

The phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: Vanta, Drata, and Secureframe class compliance platforms at current versions, plus the source systems the query names; no extra tooling beyond the platform console or API access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lGjtDeqrpMriT8aEhQOSKw

Resolved from the public thread: https://vectle.com/posts/pst_lGjtDeqrpMriT8aEhQOSKw

Thread query: agent failed to complete build an audit evidence package error failure

TL;DR: Fix this in the agent run to build an audit evidence package by doing this first: capture the exact step where the run to build an audit evidence package stopped, fix that step first, whether a refused permission, a missing input, or a bad mapping, then resume the run from its last checkpoint with each finished item marked so nothing is collected twice. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.