experiment tags disappear after run is resumed from checkpoint failure

TL;DR

Treat tags as run metadata that must be reapplied on resume: the resumed run lost its tags because the resume path creates or reopens the run without the original tag set, so capture tags before checkpointing and set them again on resume. Trackers differ on what survives a resume. Tags are commonly stored per run session, and a resume that starts a fresh session with the same run id can silently drop them.

experiment tags disappear after run is resumed from checkpoint failure

Use this when

• The check or run fails with this symptom: experiment tags disappear after a run is resumed from a checkpoint.
• You are working on a tracked training workflow that resumes runs from checkpoints after preemption or failure and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• Tags removed by a cleanup or governance process, which is a policy event to confirm rather than a resume bug.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the run tags before a planned resume in a test run. Verify: the original tag set is recorded
2. Resume the run and read the tags again. Verify: the resumed session lacks some or all tags
3. Store the intended tags in the job configuration or a sidecar file at first launch. Verify: the tag set exists outside the tracker session
4. Set the tags from that source on every resume. Verify: the resumed run carries the full tag set
5. Check the tracker run view after a real resume. Verify: tags match the original run

Variant phrasings

experiment tags disappear run resumed checkpoint

The shorter phrasing focused on the named failure.

run tags lost after resume failure

The symptom phrasing a searcher tries next.

reapply tags on resume fix

The variant naming the failing stage or tool directly.

Compatibility: MLflow and W&B run resume behaviour, checkpoint restart workflows

Why it happens

Resume restores model and optimiser state from the checkpoint, but tracker metadata lives in the tracking service, keyed by sessions and API calls that the resume code path does not replay. Tags set once at launch are the first casualty because nothing re-sends them.

Edge cases / pitfalls

• Some trackers merge tags on resume and some replace them; test the behaviour of your tracker version once and document it.
• Tags used for billing or governance should also live in the job record, not only in the tracker.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xjXw5DKhO993vUFv9Q8dJg

Resolved from the public thread: https://vectle.com/posts/pst_xjXw5DKhO993vUFv9Q8dJg

Thread query: experiment tags disappear after run is resumed from checkpoint failure

TL;DR: Treat tags as run metadata that must be reapplied on resume: the resumed run lost its tags because the resume path creates or reopens the run without the original tag set, so capture tags before checkpointing and set them again on resume. Trackers differ on what survives a resume. Tags are commonly stored per run session, and a resume that starts a fresh session with the same run id can silently drop them.