GCP budget apply fails with 403 from the Billing Budgets API - the principal needs roles/billing.costsManager on the...

TL;DR

Creating a billing budget is a billing-account-level action, and the service account or user running the apply lacks the role there. Grant roles/billing.costsManager on the billing account itself (not on a project), then re-run the apply.

Error 403: The caller does not have permission

Use this when

• terraform apply on google_billing_budget returns 403.
• gcloud billing budgets create is permission-denied.
• The same identity can manage projects but not budgets.

Not for this skill when

• The budget creates but alerts never fire. That is a notification channel problem.
• The 403 names a different API. Check that API's required role.

Steps

1. Identify the principal: the Terraform service account or the user running gcloud. Verify: you know the exact member string.
2. Grant the role on the billing account: in the console open the billing account's IAM page (not any project's IAM page) and add roles/billing.costsManager for the principal. Verify: the grant appears under the billing account's permissions.
3. Re-run the apply or the gcloud command. Verify: the 403 is gone and the budget is created.
4. If you run through a CI service account, grant it there too; user grants do not transfer. Verify: the CI apply succeeds.
5. Prefer the narrow role. roles/billing.admin also works but is broader than budget management needs. Verify: the principal has costsManager and nothing wider unless justified.

Organization policy variant

Some orgs restrict who can hold billing roles. Verify: no organization policy blocks the grant before assuming the role name is wrong.

Compatibility: GCP Billing Budgets API, Terraform Google provider.

Why it happens

Budgets live on the billing account, above projects in the hierarchy. Project-level billing roles do not reach them, so identities that are powerful inside a project are still denied at the billing account until the role is granted at the right level.

Edge cases / pitfalls

• The billing account IAM page is easy to miss; project IAM looks similar and does nothing here.
• Role propagation can take a few minutes. Retry before escalating.
• roles/billing.viewer lets you read budgets but not create them. The error looks the same.
• If budgets are managed by a different team, request the grant through them rather than widening your own roles.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstlbVLQMHoiU-adCtPYv-gw