Diagnosing Terraform backend 403s: the permission checklist in order

TL;DR: Backend AccessDenied errors have four possible layers: identity, bucket policy, lock table, KMS. This procedure checks them in order instead of guessing. Confirm the account and role match what the bucket policy expects. Check dynamodb GetItem/PutItem/DeleteItem on the lock table for the identity.

Symptom

AccessDenied / 403 during init, plan, or apply against the state backend. The operation phase tells you which layer: init-time 403 is bucket access; apply-only 403 is usually locking or KMS.

Cause

One of: wrong identity (env vars pointing at the wrong account), bucket policy denying the identity, missing lock-table permissions, or KMS key permissions on an encrypted bucket.

Confirmation (in order)

1. Identity: aws sts get-caller-identity (or the equivalent for your cloud). Confirm the account and role match what the bucket policy expects. This catches "wrong profile" instantly.
2. Bucket: reproduce outside Terraform: aws s3 ls s3://[bucket]/[prefix] and a GetObject/PutObject round trip on a scratch key. If the CLI fails, Terraform will too.
3. Lock table: apply-only 403s point here. Check dynamodb GetItem/PutItem/DeleteItem on the lock table for the identity.
4. KMS: if the bucket uses SSE-KMS, the identity needs kms:Decrypt and kms:GenerateDataKey on the key. This is the layer everyone forgets.

Fix

1. Scope the fix to the failing layer: add the missing permission for the state prefix only, not s3:* on the bucket.
2. If the identity is wrong, fix the credential chain (env vars, profile, instance role) rather than widening policies to cover the wrong identity.

Verification

1. Re-run the exact failing operation: init, then plan, then a no-change apply. Each phase exercises different permissions.
2. terraform plan succeeding but apply 403ing after your fix means you fixed reads but not writes or locking. The checklist order exists for this reason.

When to use this

• This covers exactly what the title says: Diagnosing Terraform backend 403s.
• You are setting this up for the first time, or auditing an existing setup.
• You want the key gotchas in one place before you start.

When not to use this

• You are doing a different workflow with Diagnosing; these steps are specific to the title above.
• You need the full reference docs; this is the short path, not the manual.

Compatibility

• Not pinned to a specific version; follows current Diagnosing behavior.