Diagnose and fix a broken fastai learn.lr_find() curve

TL;DR

For "CancelValidException": the fix depends on which of these failure modes you have, so classify first from the observable evidence.

CancelValidException

When to use

You are seeing this: If the curve ends abruptly, this is why. - numit is silently capped at the number of training batches: numit = min(numit, nbatch). Use this skill when you run into "Diagnose and fix a broken fastai learn.lr_find() curve".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Diagnose and fix a broken fastai learn.lr_find() curve

Use this when learn.lr_find() produces a useless plot: the curve is flat, diverges immediately, hits NaN, stops after a few iterations, or has no discernible valley, and the printed learning-rate suggestion is garbage. The fix depends on which of these failure modes you have, so classify first from the observable evidence.

0. How the LR finder actually works (fastai v2)

learn.lr_find() launches a mock training run with an exponentially growing learning rate, from start_lr=1e-7 to end_lr=10 over num_it=100 iterations:

learn.lr_find(start_lr=1e-7, end_lr=10, num_it=100, stop_div=True,
              show_plot=True, suggest_funcs=(SuggestionMethod.Valley))

Mechanics that matter for diagnosis:

• Validation is skipped during the finder (before_validate raises CancelValidException), so only training loss is recorded.

• It stops early when the smoothed loss exceeds 4x the best smoothed loss (stop_div=True). If the curve ends abruptly, this is why.

• num_it is silently capped at the number of training batches: num_it = min(num_it, n_batch). On a tiny dataset the finder runs fewer than 100 iterations with no warning.

• Your weights are restored afterwards. The finder saves the model to a temp file before the mock run and reloads it (with optimizer state) at the end. The diverged weights at the end of the plot are discarded, you can train immediately after.

• Suggestions are computed on recorder.lrs[num_it//10:-5], the first 10% and last 5 iterations are trimmed, and everything from the first NaN onward is dropped. The default suggestion method is SuggestionMethod.Valley (a point 2/3 through the longest decreasing run); Minimum (min loss / 10), Steep (steepest negative gradient w.r.t. log-lr), and Slide are the alternatives. lr_find() returns them as a named tuple, e.g. SuggestedLRs(valley=0.003).

1. Classify the failure from the plot

A. Curve stops after a handful of iterations

Observable: plot ends abruptly, far fewer than num_it points.

• If len(learn.dls.train) < 100: num_it was capped at the batch count. This is normal on small datasets, not an error, but the suggestion is computed on very few points, so treat it as a rough guess and widen the search: smaller batch size or num_it=len(learn.dls.train) won't add points (it's already at the cap); the real fix is that lr_find is unreliable here, so pick the lr by hand from the curve.

• If the dataset is large: stop_div fired, smoothed loss exceeded 4x the best. That's failure mode B with an early trigger.

B. Loss explodes from the very first iterations

Observable: loss shoots up almost immediately, no decreasing region at all.

• Cause 1: start_lr=1e-7 is already too high for this model/loss pairing (common with unnormalized inputs, very deep custom models, or a loss with a huge scale). Lower the floor: learn.lr_find(start_lr=1e-9).

• Cause 2: the data itself is broken, NaN/Inf in inputs or labels, mismatched normalization. Inspect before anything else:

learn.dls.show_batch(max_n=4)
b = learn.dls.one_batch()
print([ (x.dtype, x.min().item(), x.max().item()) for x in b ])

If inputs are raw 0 255 pixels with no normalization, or any tensor contains NaN/Inf, fix the pipeline first, no learning rate will survive it.

C. Loss goes NaN partway through

Observable: curve dives or drifts, then the line ends / suggestion ignores the tail.

• lr_find drops everything from the first NaN when computing suggestions, so a NaN tail doesn't poison the suggestion, but a NaN early in the run leaves almost no usable curve.

• NaN with a healthy-looking decreasing curve before it is the normal divergence at high lr. Only worry if NaN appears at very low lr (below ~1e-5): that means numerical instability in the model or loss, not a too-high learning rate. Check for log(0)-style ops, custom losses reducing in fp16, or zero-variance batches.

D. Curve is flat, loss never decreases at any learning rate

Observable: roughly horizontal line across the whole 1e-7 → 10 range.

• The model cannot learn from these batches at any step size. The learning rate is not the problem. Check in order:

  1. learn.dls.show_batch(), do the labels actually match the inputs?
  2. Is the loss function right for the task (CrossEntropyLossFlat for

  classification, MSELossFlat for regression)? A classification learner with a regression loss (or vice versa) gives a flat or nonsense curve.

  3. Is the model accidentally frozen? learn.opt.hypers, if every

  parameter group has requires_grad=False weights, no lr helps.

  4. Did fit get called on random/untrained weights with an architecture

  whose initialization is broken (e.g. custom head without proper init)? learn.summary() sanity-checks the architecture.

E. Curve decreases through the whole range, never diverges

Observable: loss still falling at the right edge of the plot.

• You haven't seen the valley yet, end_lr=10 was too low or num_it=100 too few for the curve to turn. Raise the ceiling: learn.lr_find(end_lr=100) or increase num_it. Pick the lr at the steepest downward slope you can now see, not at the right edge.

2. Pick the learning rate from a healthy curve

A healthy curve: loss roughly flat or slightly falling at 1e-7, a clear decreasing region, then a sharp rise (divergence). Procedure:

1. Get all four suggestions and compare, don't blindly take the default:

lrs = learn.lr_find(suggest_funcs=(SuggestionMethod.Valley,
                                   SuggestionMethod.Minimum,
                                   SuggestionMethod.Steep,
                                   SuggestionMethod.Slide))
print(lrs)

2. Rules of thumb, in order of reliability:

   • Pick the lr at the steepest downward slope of the curve (what

   Steep approximates), or about 10x below the minimum (what Minimum returns). Training at the exact minimum is unstable, the minimum sits at the edge of divergence.

   • Valley (the default) picks 2/3 through the longest decreasing run;

   it's a good starting point but verify it sits on the slope, not past the minimum.

3. Re-plot without re-running to inspect: learn.recorder.plot_lr_find(). The x-axis is log scale; the last 5 iterations are trimmed from the plot (skip_end=5).

4. Feed the chosen lr into fit_one_cycle as lr_max, the 1cycle schedule starts at lr_max/25 (div=25.) and anneals to lr_max/1e5, so the suggestion is the peak, not the starting rate:

learn.fit_one_cycle(5, lr_max=3e-3)

3. Checklist

1. Plot the curve; classify into A E above before changing anything.

2. Flat or immediate-explosion curves are data/model problems, not lr problems, inspect batches and the loss function first.

3. Remember num_it is capped at the number of training batches; short curves on small datasets are expected.

4. Compare all four suggestion methods; prefer the steepest slope or min/10, never the exact minimum.

5. lr_find() restores your weights, so it is safe to run repeatedly including once after learn.unfreeze(), when the layer groups (and therefore the right lr) have changed.

Built from the fastai v2 source: LRFinder/Learner.lrfind and the valley/slide/minimum/steep suggestion functions in fastai/callback/schedule.py (exponential lr schedule, stopdiv at 4x best smoothed loss, numit capped at batch count, tmp-save weight restoration), Recorder.plotlrfind, and recurring forums.fast.ai threads on interpreting lrfind plots.