experiment metric logging error: loss became NaN failed after epoch 3 in tracked run

TL;DR

Loss going NaN after several good epochs is a training dynamics event the tracker only recorded: find the first NaN step, then inspect what changed just before it, a learning rate schedule step, a new data shard, gradient explosion on one batch, or mixed precision overflow. Lower the peak learning rate, add gradient clipping, and in the tracker compare against a run that survived the same epoch to separate data from settings.

experiment metric logging error: loss became NaN failed after epoch 3 in tracked run

Use this when

• The run or call fails with this symptom: experiment metric logging shows loss became NaN after epoch 3 in a tracked run.
• You are working on a tracked training run whose loss becomes NaN mid-training and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• NaN present from the first step, which points at initialisation or data formatting rather than mid-training dynamics.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the tracked series and note the exact step of the first NaN and the last healthy loss value and gradient norm if logged. Verify: the failure point and preceding values are known
2. Check what the schedule and data pipeline do at that point: rate changes, new shards, or a curriculum shift. Verify: candidate triggers at that step are listed
3. Compare with a surviving run's settings at the same step to isolate the differing factor. Verify: one differing factor stands out
4. Apply the fix, typically clipping, a lower peak rate, or a data repair, and rerun past the old failure step. Verify: loss stays finite well beyond epoch 3

Variant phrasings

loss became nan after epoch 3 tracked run

The verbatim phrasing.

training loss nan mid training gradient explosion fix

The dynamics phrasing.

experiment tracker nan loss diagnose first nan step

The diagnosis phrasing.

Compatibility: Experiment tracking of training metrics across frameworks

Why it happens

NaN is absorbing: once a weight or gradient goes non-finite, every later value is NaN, so the interesting moment is the transition, not the long NaN tail. Trackers preserve the timeline that makes the transition findable, including which scheduled change coincided with it.

Edge cases / pitfalls

• Mixed precision loss scaling can hide the overflow step; the scaler's skip events in the log align with the first bad step.
• A single corrupt training example poisons its batch; if NaN recurs at the same data position across runs, suspect the data.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EcXr5deDuM-VIt8BGEZxZQ

Resolved from the public thread: https://vectle.com/posts/pst_EcXr5deDuM-VIt8BGEZxZQ

Thread query: experiment metric logging error: loss became NaN failed after epoch 3 in tracked run

TL;DR: Loss going NaN after several good epochs is a training dynamics event the tracker only recorded: find the first NaN step, then inspect what changed just before it, a learning rate schedule step, a new data shard, gradient explosion on one batch, or mixed precision overflow. Lower the peak learning rate, add gradient clipping, and in the tracker compare against a run that survived the same epoch to separate data from settings.