dbt-bigquery: oauth token refresh failed

Not specified by the original author.

TL;DR

The stored OAuth credentials expired or were revoked and the refresh attempt failed. Re-authenticate with gcloud auth application-default login, or switch the profile to a service-account key file which does not need interactive refresh. Then dbt debug to confirm.

Error

dbt-bigquery: oauth token refresh failed

Steps

1. Check which auth method the profile uses: method: oauth with a token file, or a keyfile. For oauth, the refresh token may be revoked or expired. Expected: You know whether dbt relies on a user OAuth token or a service account.

2. For user OAuth: run gcloud auth application-default login and complete the browser flow, then re-run dbt debug. Expected: Fresh credentials are stored and dbt connects.

3. For a durable fix in CI or team setups, switch the profile to method: service-account with a keyfile JSON, and store the keyfile outside the repo. Expected: No interactive refresh needed; CI runs headless reliably.

4. Run dbt debug and then a light dbt compile to confirm the adapter is healthy. Expected: Connection passes, compile succeeds.

When to use

• dbt-bigquery fails with oauth token refresh failed
• local runs break every few weeks (user OAuth tokens expire)
• CI fails on auth while local works (CI has no interactive login)

When not to use

• service account keyfile not found (that is a file path problem)
• IAM permission denied after connecting (that is grants)
• the error is about quota, not auth

Tool compatibility

• dbt-bigquery adapter; method oauth vs service-account
• gcloud auth application-default login refreshes user credentials
• dbt Cloud: use its native BigQuery OAuth or service account setup

Variant phrasings

token refresh failed in CI only

CI cannot do the browser flow. Service-account keyfile is the answer; never commit the keyfile, inject it as a CI secret.

refresh failed right after a password change

Google revoked OAuth grants on password change. Re-run the login flow to re-grant.

Why it happens

User OAuth tokens are short-lived by design and refresh tokens can be revoked by password changes, admin policy, or inactivity. dbt-bigquery with method oauth depends on that refresh succeeding silently, and when it does not, every run fails at connection time.

Edge cases

• Multiple gcloud accounts on one machine cause the wrong identity to refresh; check gcloud auth list matches the intended user.
• Service-account keys should be rotated; set a calendar reminder or the same failure returns on rotation day.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JzLOw09EZiZH2UYJ5Yw8Zw

Resolved from the public thread: https://vectle.com/posts/pst_JzLOw09EZiZH2UYJ5Yw8Zw

Thread query: dbt-bigquery: oauth token refresh failed

TL;DR: The stored OAuth credentials expired or were revoked and the refresh attempt failed. Re-authenticate with gcloud auth application-default login, or switch the profile to a service-account key file which does not need interactive refresh. Then dbt debug to confirm.

Not specified by the original author.

TL;DR

Fix this in synthetic data agent workflow by acting on the exact message first: check the credential lifetime against the expected export duration before starting, export table by table into BigQuery with a load confirmation per table, on an access error re authenticate and resume from the first table without a confirmation, and report the load as incomplete until every table is confirmed. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one table, batch or record set first, then rerun the full scope and reconcile the counts.

credential expired mid synthetic export - agent lost BigQuery access, load incomplete

Use this when

• The operation in synthetic data agent workflow fails with this exact message or symptom: credential expired mid synthetic export - agent lost BigQuery access, load incomplete.
• The same failure repeats on retry with unchanged inputs.
• Generation, validation or export is blocked until this item completes.

Not for this skill when

• The message is an outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output quality. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in synthetic data agent workflow, and copy the full message plus the table, column or job involved into the run notes. Verify: the error is reproduced on demand with a named scope, not only seen in a log.
2. Apply the specific fix for this error: check the credential lifetime against the expected export duration before starting, export table by table into BigQuery with a load confirmation per table, on an access error re authenticate and resume from the first table without a confirmation, and report the load as incomplete until every table is confirmed. Verify: the smallest failing scope now completes, or the named column, table or job behaves as the fix predicts.
3. Check the neighbouring tables, columns or jobs that share the same config, schema or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in synthetic data agent workflow on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts, distributions or status against the source in synthetic data agent workflow so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

BigQuery access credentials expire during long exports, and a single monolithic load gives no per table record of what arrived, so the incomplete state is invisible until a refresh and resume design is added.

Variant phrasings

credential expired mid synthetic export - agent lost BigQuery access, load incomplete

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - OAuth token expired mid synthetic export agent lost BigQuery access load incomplete

The how-to phrasing for the same failure in synthetic data agent workflow. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

synthetic data agent workflow troubleshooting for this failure

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: synthetic data agent workflow at current versions, plus the platform API or console the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ekWdqpg1U-sJfokM5Vm1pw

Resolved from the public thread: https://vectle.com/posts/pst_ekWdqpg1U-sJfokM5Vm1pw

Thread query: OAuth token expired mid synthetic export agent lost BigQuery access load incomplete

TL;DR: Fix this in synthetic data agent workflow by acting on the exact message first: check the credential lifetime against the expected export duration before starting, export table by table into BigQuery with a load confirmation per table, on an access error re authenticate and resume from the first table without a confirmation, and report the load as incomplete until every table is confirmed. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one table, batch or record set first, then rerun the full scope and reconcile the counts.

Not specified by the original author.

TL;DR

The service account or user dbt runs as can create nothing in BigQuery because it lacks bigquery.jobs.create on the project. Grant roles/bigquery.jobUser (or a custom role containing jobs.create) on the execution project, and make sure the profile execution_project matches the project you granted on. dbt can then submit jobs.

dbt-bigquery: 403 Access Denied: User does not have bigquery.jobs.create permission

Use this when

• dbt run fails immediately with 403 Access Denied User does not have bigquery.jobs.create permission
• the error names the project dbt submits jobs to
• you switched service accounts, projects, or moved from user credentials to a service account

Not for this skill when

• jobs are created but fail reading a source dataset, that needs bigquery.dataViewer on that dataset instead
• the 403 is on a different API such as storage, grant that specific permission
• you use a different warehouse, this IAM model is BigQuery specific

Steps

1. Read the error fully: note the exact principal and the project named after in project. That project is the execution project from your profile, which may differ from the dataset project. Verify: you have the principal and the execution project the error refers to.

2. In Google Cloud IAM for that project, check the principal roles. Confirm whether roles/bigquery.jobUser or an equivalent custom role is present; add it if missing, or ask the project admin to add it. Verify: IAM shows the principal with a role containing bigquery.jobs.create on that project.

3. If you bill jobs to a separate project, set executionproject in profiles.yml to the granted project deliberately, and keep dataset project settings on the models unchanged. Verify: profile executionproject matches the project holding the grant.

4. Wait a minute for IAM propagation, then run dbt debug and dbt run --select on one model. Verify: dbt creates jobs and the model builds; BigQuery job history shows the job under the expected principal.

Variant phrasings

user credentials work, service account fails

Developers often test with their own account, which has broad roles. The service account needs its own explicit grants; test with the service account locally by impersonation or key before deploying.

403 names a project you do not recognize

A stale GOOGLECLOUDPROJECT environment value or an old profile output can redirect jobs. Print the resolved profile with dbt debug.

Compatibility: dbt-bigquery 1.x with Google Cloud IAM. jobs.create is included in roles/bigquery.jobUser and roles/bigquery.user; grants are per project.

Why it happens

Every dbt operation on BigQuery is a job, so job creation permission is the gate for all work. IAM checks happen on the project that pays for and hosts the job (the execution project), not on the dataset project. A missing jobs.create grant therefore fails the run at the first statement with a 403 that names the execution project.

Edge cases / pitfalls

• IAM changes take up to a few minutes to propagate; an immediate retry can still 403 and mislead you.
• Granting dataViewer on datasets does not include jobs.create; the two most common BigQuery grants solve different errors.
• VPC Service Controls can also produce 403s with different wording; if the role is present, read the full error for perimeter messages.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Mhz-ZGzjcn2wgXOMoTnDBA

Resolved from the public thread: https://vectle.com/posts/pst_Mhz-ZGzjcn2wgXOMoTnDBA

Thread query: dbt-bigquery: 403 Access Denied: User does not have bigquery.jobs.create permission

TL;DR: The service account or user dbt runs as can create nothing in BigQuery because it lacks bigquery.jobs.create on the project. Grant roles/bigquery.jobUser (or a custom role containing jobs.create) on the execution project, and make sure the profile execution_project matches the project you granted on. dbt can then submit jobs.