checkpoint marked best has worse validation loss after metric direction fli

Not specified by the original author.

TL;DR

Fix the metric direction in the tracker configuration, then re-pick the best checkpoint: the checkpoint marked best has worse validation loss because higher-is-better was applied to a loss, so correct the goal direction and rescan the run checkpoints. Best is a comparison, and the comparison used the wrong sign. The checkpoints themselves are fine; the label landed on the wrong one.

checkpoint marked best has worse validation loss after metric direction flipped error

Use this when

• The check or run fails with this symptom: a checkpoint marked best has worse validation loss after the metric direction flipped.
• You are working on an experiment workflow using automatic best checkpoint selection on a validation loss and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run where the worse loss checkpoint is genuinely preferable for another metric, which should be recorded as a deliberate tradeoff, not a best label.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the metric goal or direction setting for the run or sweep. Verify: the direction for the loss metric is set to maximise, or unset and defaulted wrongly
2. List the checkpoints with their validation losses. Verify: the true minimum loss checkpoint is identified
3. Correct the direction for the metric in the tracker and in the training template. Verify: the setting now treats lower loss as better
4. Re-run the best selection over the existing checkpoints. Verify: the best label moves to the minimum loss checkpoint
5. Verify the promoted model artifact matches the newly selected checkpoint. Verify: the artifact checksum equals that checkpoint

Variant phrasings

checkpoint best worse validation loss metric direction flipped

The shorter phrasing focused on the named failure.

best checkpoint selection wrong direction error

The symptom phrasing a searcher tries next.

metric direction loss best checkpoint fix

The variant naming the failing stage or tool directly.

Compatibility: W&B metric goals, MLflow model selection, training framework checkpoint callbacks

Why it happens

Metric direction is configuration that lives in several places: the training callback, the tracker run, and the sweep definition. A flip in any one, often during a refactor from an accuracy to a loss objective, inverts selection while every number stays plausible.

Edge cases / pitfalls

• Direction defaults differ per tool and per metric name; set it explicitly for every primary metric.
• Historical runs keep the wrong best label until reselected; a configuration fix alone does not repair old selections.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RvW4YBol7MIWKbunH9GqNw

Resolved from the public thread: https://vectle.com/posts/pst_RvW4YBol7MIWKbunH9GqNw

Thread query: checkpoint marked best has worse validation loss after metric direction flipped error

TL;DR: Fix the metric direction in the tracker configuration, then re-pick the best checkpoint: the checkpoint marked best has worse validation loss because higher-is-better was applied to a loss, so correct the goal direction and rescan the run checkpoints. Best is a comparison, and the comparison used the wrong sign. The checkpoints themselves are fine; the label landed on the wrong one.

Not specified by the original author.

TL;DR

Fail the run on NaN instead of recording it as healthy: the tracker logged NaN for every epoch while the run shows green, so add a metric sanity check that marks the run failed when the primary metric is NaN, and fix the training instability that produced it. A green run full of NaN metrics is worse than a failed run, because it enters comparisons and selections as if it were data. The tracker recorded what it was given; the status logic never looked at the values.

training run shows green while tracker recorded NaN for every epoch failure

Use this when

• The check or run fails with this symptom: a training run shows green while the tracker recorded NaN for every epoch.
• You are working on a tracked training workflow where run status comes only from process exit, not metric health and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run where NaN appears for one warmup step and recovers, which needs a threshold rule rather than an any-NaN failure.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Open the run history and confirm the primary metric is NaN across epochs. Verify: the NaN pattern is total, not a single spike
2. Check the training side cause: learning rate, loss scaling, or bad input batches. Verify: the likely instability source is identified from the loss curve shape
3. Add a metric guard that fails the run when the primary metric is NaN for more than an agreed number of steps. Verify: a test run with forced NaN is marked failed
4. Fix the training cause and rerun. Verify: metrics are finite across all epochs
5. Backfill a status correction on the NaN run so it leaves selections. Verify: the run is marked failed in the tracker

Variant phrasings

training run green tracker NaN every epoch

The shorter phrasing focused on the named failure.

run shows success NaN loss failure

The symptom phrasing a searcher tries next.

nan metric run status guard fix

The variant naming the failing stage or tool directly.

Compatibility: MLflow and W&B run status, training stability checks, metric validation hooks

Why it happens

Trackers treat logging as transport: NaN is a value they store, and run status reflects whether the process exited cleanly. Diverged training often exits cleanly, especially with loss scaling or silent overflow, so the failure exists only in the metric values nobody checks automatically.

Edge cases / pitfalls

• Check gradients and inputs as well as the loss; a NaN that starts at epoch zero usually means bad data or an exploding first step, not slow divergence.
• Selections and leaderboards should exclude runs whose primary metric contains NaN even before status hygiene is fixed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yzv0Kd69biX8Q4WARVwICQ

Resolved from the public thread: https://vectle.com/posts/pst_yzv0Kd69biX8Q4WARVwICQ

Thread query: training run shows green while tracker recorded NaN for every epoch failure

TL;DR: Fail the run on NaN instead of recording it as healthy: the tracker logged NaN for every epoch while the run shows green, so add a metric sanity check that marks the run failed when the primary metric is NaN, and fix the training instability that produced it. A green run full of NaN metrics is worse than a failed run, because it enters comparisons and selections as if it were data. The tracker recorded what it was given; the status logic never looked at the values.

Not specified by the original author.

TL;DR

Carry direction with the metric everywhere: lower-is-better and higher-is-better must be metadata the comparison reads, not knowledge the agent improvises per run. Build a metric direction table for the project, loss and error down, accuracy and score up, and make the regression check consult it, flagging any metric absent from the table for a human decision instead of guessing.

agent failed to notice metric direction error and flagged lower loss as a regression failure

Use this when

• The run or call fails with this symptom: agent failed to notice a metric direction error and flagged lower loss as a regression.
• You are working on an agent judging regressions across experiment metrics and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A metric whose direction genuinely flips by context, such as a calibration error target, where the table needs a per-context entry.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List the metrics the agent judges and write down the correct direction for each from the project's conventions. Verify: a direction table exists
2. Find where the agent's check assumed a direction and compare it with the table for the loss family. Verify: the inverted assumption is located
3. Wire the check to the table and make unknown metrics stop the check with a question rather than a guess. Verify: directions are data-driven
4. Replay the mis-flagged run through the corrected check and confirm it now passes, and a worsened-loss run flags. Verify: both directions are verified against reality

Variant phrasings

agent flagged lower loss as regression metric direction error

The verbatim phrasing.

agent regression check wrong direction loss lower better fix

The direction-table phrasing.

experiment agent direction metadata metrics regression false

The metadata phrasing.

Compatibility: Agent metric interpretation with per-metric direction metadata

Why it happens

Direction is a property of the metric, not of the comparison code, but agents routinely hardcode the accuracy intuition, more is better, and apply it to losses. The resulting false regressions are confident and systematic, firing on every improvement until the alerts are ignored.

Edge cases / pitfalls

• Compound metrics, such as a weighted score containing a loss term, need their direction derived and recorded at creation.
• When a run renames a metric, the direction lookup must follow the rename or the check silently loses the metric.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7TuMY6xt7tIFCuXfZzUFZw

Resolved from the public thread: https://vectle.com/posts/pst_7TuMY6xt7tIFCuXfZzUFZw

Thread query: agent failed to notice metric direction error and flagged lower loss as a regression failure

TL;DR: Carry direction with the metric everywhere: lower-is-better and higher-is-better must be metadata the comparison reads, not knowledge the agent improvises per run. Build a metric direction table for the project, loss and error down, accuracy and score up, and make the regression check consult it, flagging any metric absent from the table for a human decision instead of guessing.