checkpoint marked best has worse validation loss after metric direction flipped error

TL;DR

Fix the metric direction in the tracker configuration, then re-pick the best checkpoint: the checkpoint marked best has worse validation loss because higher-is-better was applied to a loss, so correct the goal direction and rescan the run checkpoints. Best is a comparison, and the comparison used the wrong sign. The checkpoints themselves are fine; the label landed on the wrong one.

checkpoint marked best has worse validation loss after metric direction flipped error

Use this when

• The check or run fails with this symptom: a checkpoint marked best has worse validation loss after the metric direction flipped.
• You are working on an experiment workflow using automatic best checkpoint selection on a validation loss and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run where the worse loss checkpoint is genuinely preferable for another metric, which should be recorded as a deliberate tradeoff, not a best label.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the metric goal or direction setting for the run or sweep. Verify: the direction for the loss metric is set to maximise, or unset and defaulted wrongly
2. List the checkpoints with their validation losses. Verify: the true minimum loss checkpoint is identified
3. Correct the direction for the metric in the tracker and in the training template. Verify: the setting now treats lower loss as better
4. Re-run the best selection over the existing checkpoints. Verify: the best label moves to the minimum loss checkpoint
5. Verify the promoted model artifact matches the newly selected checkpoint. Verify: the artifact checksum equals that checkpoint

Variant phrasings

checkpoint best worse validation loss metric direction flipped

The shorter phrasing focused on the named failure.

best checkpoint selection wrong direction error

The symptom phrasing a searcher tries next.

metric direction loss best checkpoint fix

The variant naming the failing stage or tool directly.

Compatibility: W&B metric goals, MLflow model selection, training framework checkpoint callbacks

Why it happens

Metric direction is configuration that lives in several places: the training callback, the tracker run, and the sweep definition. A flip in any one, often during a refactor from an accuracy to a loss objective, inverts selection while every number stays plausible.

Edge cases / pitfalls

• Direction defaults differ per tool and per metric name; set it explicitly for every primary metric.
• Historical runs keep the wrong best label until reselected; a configuration fix alone does not repair old selections.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RvW4YBol7MIWKbunH9GqNw

Resolved from the public thread: https://vectle.com/posts/pst_RvW4YBol7MIWKbunH9GqNw

Thread query: checkpoint marked best has worse validation loss after metric direction flipped error

TL;DR: Fix the metric direction in the tracker configuration, then re-pick the best checkpoint: the checkpoint marked best has worse validation loss because higher-is-better was applied to a loss, so correct the goal direction and rescan the run checkpoints. Best is a comparison, and the comparison used the wrong sign. The checkpoints themselves are fine; the label landed on the wrong one.