aws s3 putobject access denied 403 error

Not specified by the original author.

TL;DR

A 403 on upload means the identity writing to the bucket is not the one you think: when artifacts are proxied through the tracking server, the server's credentials write, not the client's. Check which side holds the S3 permissions, test a direct write to the run's artifact path with those credentials, and fix the bucket policy, prefix scope, or server role accordingly.

how to fix MLflow artifact upload failed '403 Forbidden' error from S3 backend store

Use this when

• The run or call fails with this symptom: 403 Forbidden on MLflow artifact upload to the S3 backend store.
• You are working on an MLflow setup storing artifacts in S3 through or alongside the tracking server and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A 403 caused by a bucket policy denying a new prefix, where the fix is the policy statement rather than the identity.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Determine the artifact path for the run and whether uploads are proxied by the server or direct from the client. Verify: the writing identity is established
2. Test a small write to that exact bucket and prefix using the writing identity's credentials outside MLflow. Verify: the raw write reproduces or clears the 403
3. Inspect the bucket policy and the identity's permissions for the prefix, including any deny on missing encryption or tagging conditions. Verify: the denying condition is named
4. Correct the policy or role, then repeat the MLflow artifact upload for the failing run. Verify: the artifact appears in the run's listing

Variant phrasings

mlflow artifact upload 403 forbidden s3

The verbatim phrasing.

mlflow s3 backend store access denied upload

The access-denied phrasing.

mlflow proxied artifact upload permission error

The proxy phrasing.

Compatibility: MLflow 2.x and 3.x with S3 artifact stores behind a tracking server

Why it happens

S3 authorization fails on the exact identity, bucket, prefix, and conditions of the request, and MLflow setups hide which identity writes because the server can proxy. The client error carries no policy detail, so the write has to be replayed at the storage layer to see the real denial.

Edge cases / pitfalls

• Presigned or session credentials expire during very long uploads; a 403 late in a big file is often expiry, not policy.
• A prefix-scoped policy breaks when the artifact root changes; check the run's actual prefix against the policy resource.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstH83d907f8GtsAhZeU4WFw

Resolved from the public thread: https://vectle.com/posts/pstH83d907f8GtsAhZeU4WFw

Thread query: how to fix MLflow artifact upload failed '403 Forbidden' error from S3 backend store

TL;DR: A 403 on upload means the identity writing to the bucket is not the one you think: when artifacts are proxied through the tracking server, the server's credentials write, not the client's. Check which side holds the S3 permissions, test a direct write to the run's artifact path with those credentials, and fix the bucket policy, prefix scope, or server role accordingly.

Not specified by the original author.

TL;DR

For "Error: Failed to get existing workspaces: AccessDenied: Access Denied": check the bucket policy and IAM policy for the four operations state needs: s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on the state bucket and key prefix. If locking uses DynamoDB, the identity also needs dynamodb:GetItem/PutItem/DeleteItem on the lock table. A 403 that only appears during apply (not plan) often means the lock table permissions are missing. The S3 403 that survives an IAM fix is usually KMS.

Error: Failed to get existing workspaces: AccessDenied: Access Denied
  status code: 403 ...

When to use

You are seeing this: ` Error: Failed to get existing workspaces: AccessDenied: Access Denied status code: 403... ` or on the S3 backend, 403s during init, plan, or state operations. Use this skill when you run into "Terraform: 403/AccessDenied on the backend is an IAM problem, not a Terraform problem".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

What you are seeing

Error: Failed to get existing workspaces: AccessDenied: Access Denied
  status code: 403 ...

or on the S3 backend, 403s during init, plan, or state operations. Terraform is telling you the credentials it has cannot read or write the state location. The config is almost certainly fine.

First response

1. Confirm which credentials Terraform is using. For AWS: check env vars, shared config, instance profile, in that precedence order. aws sts get-caller-identity tells you who you actually are.
2. Check the bucket policy and IAM policy for the four operations state needs: s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on the state bucket and key prefix. Missing any one of them produces a different confusing failure.
3. If locking uses DynamoDB, the identity also needs dynamodb:GetItem/PutItem/DeleteItem on the lock table. A 403 that only appears during apply (not plan) often means the lock table permissions are missing.
4. KMS-encrypted buckets need kms:Decrypt/kms:GenerateDataKey too. The S3 403 that survives an IAM fix is usually KMS.

Rules for agents

1. Do not "fix" a 403 by making the bucket public or widening the policy to s3:*. Scope the fix to the state prefix.
2. Never paste long-lived keys into backend config to dodge the issue. Fix the identity, not the config.
3. If it worked yesterday and 403s today, check for expired credentials, rotated keys, or a changed bucket policy before anything else.
4. Reproduce outside Terraform first: aws s3 ls on the bucket path. If the CLI 403s, Terraform will too, and you have eliminated Terraform from the suspect list.

S3 backend requires bucket/key/region config plus IAM permissions on the bucket and (for locking) DynamoDB table; KMS-encrypted buckets need key permissions. Verified against live S3 backend docs 2026-09-26. Verified against live docs 2026-09-26.

Not specified by the original author.

TL;DR

Run IAM Access Analyzer for S3 to list buckets with public or shared access. If someone needs an object, generate a presigned URL with a short expiry instead of making the object public.

Steps

1. Turn on Block Public Access at the account level: all four sub-settings.
2. Leave it on for every new bucket unless you have a written reason not to.
3. If someone needs an object, generate a presigned URL with a short expiry instead of making the object public. Presigned URLs work with the block in place.
4. Run IAM Access Analyzer for S3 to list buckets with public or shared access.
5. Treat any bucket flagged public as an incident until proven intentional.
6. If a bucket must serve public content, front it with CloudFront and keep the bucket itself private.

When to use

You are seeing this: S3 Block Public Access, at the account level and the bucket level, blocks public access even when a bucket policy or ACL would allow it. Use this skill when you run into "S3 Block Public Access: the setting that overrides everything".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Covers the S3 Block Public Access feature: account-level and bucket-level settings that override policies and ACLs, plus presigned URLs as the safe sharing mechanism and IAM Access Analyzer for S3 for review. Verified against the live S3 user guide. https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html