Determined AI experiment failed error "checkpoint storage config missing bucket"

Not specified by the original author.

TL;DR

Configure the checkpoint storage before launching: Determined AI refuses to start the experiment because its checkpoint storage has no bucket, so set the checkpoint storage configuration for the cluster or the experiment, pointing at a bucket the master can write to. Checkpoints are written throughout training, so Determined validates the storage configuration at submission. A missing bucket is a configuration gap, not a runtime storage failure.

Determined AI experiment failed error "checkpoint storage config missing bucket"

Use this when

• The check or run fails with this symptom: a Determined AI experiment fails with checkpoint storage config missing bucket.
• You are working on a Determined AI deployment using object storage for experiment checkpoints and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A deployment configured for shared filesystem checkpoints, where the bucket message means the storage type was changed and the experiment config must match it.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Open the effective experiment configuration and the cluster checkpoint storage section. Verify: the storage type and its fields are visible, with the bucket empty
2. Create or identify the bucket for checkpoints and confirm the master service account can write to it. Verify: a test write to the bucket succeeds
3. Set the bucket, and any prefix, in the cluster configuration or per experiment. Verify: the configuration shows the bucket in place
4. Submit a short experiment that checkpoints once. Verify: the checkpoint appears in the bucket
5. Set the configuration at cluster level so future experiments inherit it. Verify: a second experiment starts without the error

Variant phrasings

determined ai checkpoint storage config missing bucket

The shorter phrasing focused on the named failure.

determined experiment failed checkpoint storage error

The symptom phrasing a searcher tries next.

determined checkpoint bucket config fix

The variant naming the failing stage or tool directly.

Compatibility: Determined AI experiment configuration, checkpoint storage for S3, GCS, Azure, and shared filesystems

Why it happens

Determined separates checkpoint storage from the experiment definition, and templates copied between clusters often carry the storage type without the destination. Submission validation catches the missing bucket early, before any GPU time is spent.

Edge cases / pitfalls

• Per-experiment overrides silently shadow cluster settings; check both levels when the error persists after a cluster fix.
• Checkpoint buckets accumulate large objects; set a lifecycle policy alongside the bucket, not months later.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_V-R52xFoFcHxpfAqKkmXsw

Resolved from the public thread: https://vectle.com/posts/pst_V-R52xFoFcHxpfAqKkmXsw

Thread query: Determined AI experiment failed error "checkpoint storage config missing bucket"

TL;DR: Configure the checkpoint storage before launching: Determined AI refuses to start the experiment because its checkpoint storage has no bucket, so set the checkpoint storage configuration for the cluster or the experiment, pointing at a bucket the master can write to. Checkpoints are written throughout training, so Determined validates the storage configuration at submission. A missing bucket is a configuration gap, not a runtime storage failure.