training run shows green while tracker recorded NaN for every epoch failure

TL;DR

Fail the run on NaN instead of recording it as healthy: the tracker logged NaN for every epoch while the run shows green, so add a metric sanity check that marks the run failed when the primary metric is NaN, and fix the training instability that produced it. A green run full of NaN metrics is worse than a failed run, because it enters comparisons and selections as if it were data. The tracker recorded what it was given; the status logic never looked at the values.

training run shows green while tracker recorded NaN for every epoch failure

Use this when

• The check or run fails with this symptom: a training run shows green while the tracker recorded NaN for every epoch.
• You are working on a tracked training workflow where run status comes only from process exit, not metric health and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run where NaN appears for one warmup step and recovers, which needs a threshold rule rather than an any-NaN failure.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Open the run history and confirm the primary metric is NaN across epochs. Verify: the NaN pattern is total, not a single spike
2. Check the training side cause: learning rate, loss scaling, or bad input batches. Verify: the likely instability source is identified from the loss curve shape
3. Add a metric guard that fails the run when the primary metric is NaN for more than an agreed number of steps. Verify: a test run with forced NaN is marked failed
4. Fix the training cause and rerun. Verify: metrics are finite across all epochs
5. Backfill a status correction on the NaN run so it leaves selections. Verify: the run is marked failed in the tracker

Variant phrasings

training run green tracker NaN every epoch

The shorter phrasing focused on the named failure.

run shows success NaN loss failure

The symptom phrasing a searcher tries next.

nan metric run status guard fix

The variant naming the failing stage or tool directly.

Compatibility: MLflow and W&B run status, training stability checks, metric validation hooks

Why it happens

Trackers treat logging as transport: NaN is a value they store, and run status reflects whether the process exited cleanly. Diverged training often exits cleanly, especially with loss scaling or silent overflow, so the failure exists only in the metric values nobody checks automatically.

Edge cases / pitfalls

• Check gradients and inputs as well as the loss; a NaN that starts at epoch zero usually means bad data or an exploding first step, not slow divergence.
• Selections and leaderboards should exclude runs whose primary metric contains NaN even before status hygiene is fixed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yzv0Kd69biX8Q4WARVwICQ

Resolved from the public thread: https://vectle.com/posts/pst_yzv0Kd69biX8Q4WARVwICQ

Thread query: training run shows green while tracker recorded NaN for every epoch failure

TL;DR: Fail the run on NaN instead of recording it as healthy: the tracker logged NaN for every epoch while the run shows green, so add a metric sanity check that marks the run failed when the primary metric is NaN, and fix the training instability that produced it. A green run full of NaN metrics is worse than a failed run, because it enters comparisons and selections as if it were data. The tracker recorded what it was given; the status logic never looked at the values.