Sacred experiment failed error "FailedRun: database observer could not store run"

Not specified by the original author.

TL;DR

Fix the observer connection, because the run itself already failed to be stored: Sacred records runs through its database observer, so check the observer database is reachable and accepting writes, then rerun, since an unstored run cannot be recovered after the fact. Sacred separates running the experiment from storing it. When the database observer cannot store, the experiment may compute normally and still end as a FailedRun with nothing in the database.

Sacred experiment failed error "FailedRun: database observer could not store run"

Use this when

• The check or run fails with this symptom: a Sacred experiment fails with FailedRun database observer could not store run.
• You are working on a Sacred experiment configured with a MongoDB or SQL database observer and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A file observer setup, where storage is local and a database error points at a misconfigured observer choice.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the full error for the underlying database exception. Verify: the storage failure cause is named, such as a connection refusal or an authentication failure
2. Connect to the observer database directly with the same settings the experiment uses. Verify: the database accepts a test write, or the failure reproduces
3. Fix the database side: start it, correct the connection settings, or restore the write permission. Verify: a test observer write succeeds
4. Rerun the experiment with a tiny configuration. Verify: the run is stored and queryable in the observer database
5. Add a startup check that the observer can store before training begins. Verify: a storage failure now aborts the run in seconds, not after training

Variant phrasings

sacred FailedRun database observer could not store run

The shorter phrasing focused on the named failure.

sacred experiment failed observer store error

The symptom phrasing a searcher tries next.

sacred database observer fix

The variant naming the failing stage or tool directly.

Compatibility: Sacred with MongoObserver or SqlObserver, experiment storage backends

Why it happens

Sacred observers write run documents as the experiment progresses and finalise them at the end. A database that is down, full, or reached with stale credentials turns the finalise step into a failure that marks the whole run failed, even though the failure is in bookkeeping, not in the experiment.

Edge cases / pitfalls

• The run heartbeat and the final store are different writes; a run can appear alive and still fail to store at the end.
• File observers are the fallback for environments where the database is unreliable, at the cost of central querying.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dvB3tdxQhhKeyL-Q8T-9mw

Resolved from the public thread: https://vectle.com/posts/pst_dvB3tdxQhhKeyL-Q8T-9mw

Thread query: Sacred experiment failed error "FailedRun: database observer could not store run"

TL;DR: Fix the observer connection, because the run itself already failed to be stored: Sacred records runs through its database observer, so check the observer database is reachable and accepting writes, then rerun, since an unstored run cannot be recovered after the fact. Sacred separates running the experiment from storing it. When the database observer cannot store, the experiment may compute normally and still end as a FailedRun with nothing in the database.