# Diagnose and fix a broken fastai `learn.lr_find()` curve
Use this when `learn.lr_find()` produces a useless plot: the curve is flat,
diverges immediately, hits NaN, stops after a few iterations, or has no
discernible valley — and the printed learning-rate suggestion is garbage.
The fix depends on which of these failure modes you have, so classify first
from the observable evidence.
## 0. How the LR finder actually works (fastai v2)
`learn.lr_find()` launches a *mock* training run with an exponentially
growing learning rate, from `start_lr=1e-7` to `end_lr=10` over `num_it=100`
iterations:
```python
learn.lr_find(start_lr=1e-7, end_lr=10, num_it=100, stop_div=True,
show_plot=True, suggest_funcs=(SuggestionMethod.Valley))
```
Mechanics that matter for diagnosis:
- **Validation is skipped** during the finder (`before_validate` raises
`CancelValidException`), so only training loss is recorded.
- It **stops early when the smoothed loss exceeds 4x the best smoothed loss**
(`stop_div=True`). If the curve ends abruptly, this is why.
- **`num_it` is silently capped at the number of training batches**:
`num_it = min(num_it, n_batch)`. On a tiny dataset the finder runs fewer
than 100 iterations with no warning.
- **Your weights are restored afterwards.** The finder saves the model to a
temp file before the mock run and reloads it (with optimizer state) at the
end. The diverged weights at the end of the plot are discarded — you can
train immediately after.
- Suggestions are computed on `recorder.lrs[num_it//10:-5]` — the first 10%
and last 5 iterations are trimmed, and everything from the first NaN
onward is dropped. The default suggestion method is `SuggestionMethod.Valley`
(a point 2/3 through the longest decreasing run); `Minimum` (min loss / 10),
`Steep` (steepest negative gradient w.r.t. log-lr), and `Slide` are the
alternatives. `lr_find()` returns them as a named tuple, e.g.
`SuggestedLRs(valley=0.003)`.
## 1. Classify the failure from the plot
### A. Curve stops after a handful of iterations
Observable: plot ends abruptly, far fewer than `num_it` points.
- If `len(learn.dls.train)` < 100: `num_it` was capped at the batch count.
This is normal on small datasets, not an error — but the suggestion is
computed on very few points, so treat it as a rough guess and widen the
search: smaller batch size or `num_it=len(learn.dls.train)` won't add
points (it's already at the cap); the real fix is that lr_find is
unreliable here, so pick the lr by hand from the curve.
- If the dataset is large: `stop_div` fired — smoothed loss exceeded 4x the
best. That's failure mode B with an early trigger.
### B. Loss explodes from the very first iterations
Observable: loss shoots up almost immediately, no decreasing region at all.
- Cause 1: `start_lr=1e-7` is already too high for this model/loss pairing
(common with unnormalized inputs, very deep custom models, or a loss with
a huge scale). Lower the floor: `learn.lr_find(start_lr=1e-9)`.
- Cause 2: the data itself is broken — NaN/Inf in inputs or labels,
mismatched normalization. Inspect before anything else:
```python
learn.dls.show_batch(max_n=4)
b = learn.dls.one_batch()
print([ (x.dtype, x.min().item(), x.max().item()) for x in b ])
```
If inputs are raw 0–255 pixels with no normalization, or any tensor contains
NaN/Inf, fix the pipeline first — no learning rate will survive it.
### C. Loss goes NaN partway through
Observable: curve dives or drifts, then the line ends / suggestion ignores
the tail.
- `lr_find` drops everything from the first NaN when computing suggestions,
so a NaN tail doesn't poison the suggestion — but a NaN *early* in the run
leaves almost no usable curve.
- NaN with a healthy-looking decreasing curve before it is the normal
divergence at high lr. Only worry if NaN appears at very low lr
(below ~1e-5): that means numerical instability in the model or loss,
not a too-high learning rate. Check for `log(0)`-style ops, custom losses
reducing in fp16, or zero-variance batches.
### D. Curve is flat — loss never decreases at any learning rate
Observable: roughly horizontal line across the whole 1e-7 → 10 range.
- The model cannot learn from these batches at any step size. The learning
rate is not the problem. Check in order:
1. `learn.dls.show_batch()` — do the labels actually match the inputs?
2. Is the loss function right for the task (`CrossEntropyLossFlat` for
classification, `MSELossFlat` for regression)? A classification
learner with a regression loss (or vice versa) gives a flat or
nonsense curve.
3. Is the model accidentally frozen? `learn.opt.hypers` — if every
parameter group has `requires_grad=False` weights, no lr helps.
4. Did `fit` get called on random/untrained weights with an architecture
whose initialization is broken (e.g. custom head without proper init)?
`learn.summary()` sanity-checks the architecture.
### E. Curve decreases through the whole range, never diverges
Observable: loss still falling at the right edge of the plot.
- You haven't seen the valley yet — `end_lr=10` was too low or `num_it=100`
too few for the curve to turn. Raise the ceiling:
`learn.lr_find(end_lr=100)` or increase `num_it`. Pick the lr at the
steepest downward slope you can now see, not at the right edge.
## 2. Pick the learning rate from a healthy curve
A healthy curve: loss roughly flat or slightly falling at 1e-7, a clear
decreasing region, then a sharp rise (divergence). Procedure:
1. Get all four suggestions and compare, don't blindly take the default:
```python
lrs = learn.lr_find(suggest_funcs=(SuggestionMethod.Valley,
SuggestionMethod.Minimum,
SuggestionMethod.Steep,
SuggestionMethod.Slide))
print(lrs)
```
2. Rules of thumb, in order of reliability:
- Pick the lr at the **steepest downward slope** of the curve (what
`Steep` approximates), or about **10x below the minimum** (what
`Minimum` returns). Training at the exact minimum is unstable — the
minimum sits at the edge of divergence.
- `Valley` (the default) picks 2/3 through the longest decreasing run;
it's a good starting point but verify it sits on the slope, not past
the minimum.
3. Re-plot without re-running to inspect: `learn.recorder.plot_lr_find()`.
The x-axis is log scale; the last 5 iterations are trimmed from the
plot (`skip_end=5`).
4. Feed the chosen lr into `fit_one_cycle` as `lr_max` — the 1cycle schedule
starts at `lr_max/25` (`div=25.`) and anneals to `lr_max/1e5`, so the
suggestion is the *peak*, not the starting rate:
```python
learn.fit_one_cycle(5, lr_max=3e-3)
```
## 3. Checklist
1. Plot the curve; classify into A–E above before changing anything.
2. Flat or immediate-explosion curves are data/model problems, not lr
problems — inspect batches and the loss function first.
3. Remember `num_it` is capped at the number of training batches; short
curves on small datasets are expected.
4. Compare all four suggestion methods; prefer the steepest slope or
min/10, never the exact minimum.
5. `lr_find()` restores your weights, so it is safe to run repeatedly —
including once after `learn.unfreeze()`, when the layer groups (and
therefore the right lr) have changed.