# Diagnose and fix a broken fastai learn.lr_find() curve

Diagnose a useless fastai learning-rate-finder plot by classifying the curve shape (early stop_div abort, num_it capped by batch count, NaN tail, flat curve, immediate explosion, no valley) and apply the matching fix; then pick the training lr from the steepest slope or min/10 using SuggestionMethod Valley/Minimum/Steep/Slide.

Exact reference: {"kind":"skill_version","skill_id":"skl_3_p-tcs54hBa1zl_aoImvQ","version_id":"skv_FSAJK1pL2xWzX0h3wntdNw"}

Applicability: [{"constraint":">=2.0","technology":"fastai","version_scheme":"semver"}]

# Diagnose and fix a broken fastai `learn.lr_find()` curve

Use this when `learn.lr_find()` produces a useless plot: the curve is flat,
diverges immediately, hits NaN, stops after a few iterations, or has no
discernible valley — and the printed learning-rate suggestion is garbage.
The fix depends on which of these failure modes you have, so classify first
from the observable evidence.

## 0. How the LR finder actually works (fastai v2)

`learn.lr_find()` launches a *mock* training run with an exponentially
growing learning rate, from `start_lr=1e-7` to `end_lr=10` over `num_it=100`
iterations:

```python
learn.lr_find(start_lr=1e-7, end_lr=10, num_it=100, stop_div=True,
              show_plot=True, suggest_funcs=(SuggestionMethod.Valley))
```

Mechanics that matter for diagnosis:

- **Validation is skipped** during the finder (`before_validate` raises
  `CancelValidException`), so only training loss is recorded.
- It **stops early when the smoothed loss exceeds 4x the best smoothed loss**
  (`stop_div=True`). If the curve ends abruptly, this is why.
- **`num_it` is silently capped at the number of training batches**:
  `num_it = min(num_it, n_batch)`. On a tiny dataset the finder runs fewer
  than 100 iterations with no warning.
- **Your weights are restored afterwards.** The finder saves the model to a
  temp file before the mock run and reloads it (with optimizer state) at the
  end. The diverged weights at the end of the plot are discarded — you can
  train immediately after.
- Suggestions are computed on `recorder.lrs[num_it//10:-5]` — the first 10%
  and last 5 iterations are trimmed, and everything from the first NaN
  onward is dropped. The default suggestion method is `SuggestionMethod.Valley`
  (a point 2/3 through the longest decreasing run); `Minimum` (min loss / 10),
  `Steep` (steepest negative gradient w.r.t. log-lr), and `Slide` are the
  alternatives. `lr_find()` returns them as a named tuple, e.g.
  `SuggestedLRs(valley=0.003)`.

## 1. Classify the failure from the plot

### A. Curve stops after a handful of iterations

Observable: plot ends abruptly, far fewer than `num_it` points.

- If `len(learn.dls.train)` < 100: `num_it` was capped at the batch count.
  This is normal on small datasets, not an error — but the suggestion is
  computed on very few points, so treat it as a rough guess and widen the
  search: smaller batch size or `num_it=len(learn.dls.train)` won't add
  points (it's already at the cap); the real fix is that lr_find is
  unreliable here, so pick the lr by hand from the curve.
- If the dataset is large: `stop_div` fired — smoothed loss exceeded 4x the
  best. That's failure mode B with an early trigger.

### B. Loss explodes from the very first iterations

Observable: loss shoots up almost immediately, no decreasing region at all.

- Cause 1: `start_lr=1e-7` is already too high for this model/loss pairing
  (common with unnormalized inputs, very deep custom models, or a loss with
  a huge scale). Lower the floor: `learn.lr_find(start_lr=1e-9)`.
- Cause 2: the data itself is broken — NaN/Inf in inputs or labels,
  mismatched normalization. Inspect before anything else:

```python
learn.dls.show_batch(max_n=4)
b = learn.dls.one_batch()
print([ (x.dtype, x.min().item(), x.max().item()) for x in b ])
```

If inputs are raw 0–255 pixels with no normalization, or any tensor contains
NaN/Inf, fix the pipeline first — no learning rate will survive it.

### C. Loss goes NaN partway through

Observable: curve dives or drifts, then the line ends / suggestion ignores
the tail.

- `lr_find` drops everything from the first NaN when computing suggestions,
  so a NaN tail doesn't poison the suggestion — but a NaN *early* in the run
  leaves almost no usable curve.
- NaN with a healthy-looking decreasing curve before it is the normal
  divergence at high lr. Only worry if NaN appears at very low lr
  (below ~1e-5): that means numerical instability in the model or loss,
  not a too-high learning rate. Check for `log(0)`-style ops, custom losses
  reducing in fp16, or zero-variance batches.

### D. Curve is flat — loss never decreases at any learning rate

Observable: roughly horizontal line across the whole 1e-7 → 10 range.

- The model cannot learn from these batches at any step size. The learning
  rate is not the problem. Check in order:
  1. `learn.dls.show_batch()` — do the labels actually match the inputs?
  2. Is the loss function right for the task (`CrossEntropyLossFlat` for
     classification, `MSELossFlat` for regression)? A classification
     learner with a regression loss (or vice versa) gives a flat or
     nonsense curve.
  3. Is the model accidentally frozen? `learn.opt.hypers` — if every
     parameter group has `requires_grad=False` weights, no lr helps.
  4. Did `fit` get called on random/untrained weights with an architecture
     whose initialization is broken (e.g. custom head without proper init)?
     `learn.summary()` sanity-checks the architecture.

### E. Curve decreases through the whole range, never diverges

Observable: loss still falling at the right edge of the plot.

- You haven't seen the valley yet — `end_lr=10` was too low or `num_it=100`
  too few for the curve to turn. Raise the ceiling:
  `learn.lr_find(end_lr=100)` or increase `num_it`. Pick the lr at the
  steepest downward slope you can now see, not at the right edge.

## 2. Pick the learning rate from a healthy curve

A healthy curve: loss roughly flat or slightly falling at 1e-7, a clear
decreasing region, then a sharp rise (divergence). Procedure:

1. Get all four suggestions and compare, don't blindly take the default:

```python
lrs = learn.lr_find(suggest_funcs=(SuggestionMethod.Valley,
                                   SuggestionMethod.Minimum,
                                   SuggestionMethod.Steep,
                                   SuggestionMethod.Slide))
print(lrs)
```

2. Rules of thumb, in order of reliability:
   - Pick the lr at the **steepest downward slope** of the curve (what
     `Steep` approximates), or about **10x below the minimum** (what
     `Minimum` returns). Training at the exact minimum is unstable — the
     minimum sits at the edge of divergence.
   - `Valley` (the default) picks 2/3 through the longest decreasing run;
     it's a good starting point but verify it sits on the slope, not past
     the minimum.
3. Re-plot without re-running to inspect: `learn.recorder.plot_lr_find()`.
   The x-axis is log scale; the last 5 iterations are trimmed from the
   plot (`skip_end=5`).
4. Feed the chosen lr into `fit_one_cycle` as `lr_max` — the 1cycle schedule
   starts at `lr_max/25` (`div=25.`) and anneals to `lr_max/1e5`, so the
   suggestion is the *peak*, not the starting rate:

```python
learn.fit_one_cycle(5, lr_max=3e-3)
```

## 3. Checklist

1. Plot the curve; classify into A–E above before changing anything.
2. Flat or immediate-explosion curves are data/model problems, not lr
   problems — inspect batches and the loss function first.
3. Remember `num_it` is capped at the number of training batches; short
   curves on small datasets are expected.
4. Compare all four suggestion methods; prefer the steepest slope or
   min/10, never the exact minimum.
5. `lr_find()` restores your weights, so it is safe to run repeatedly —
   including once after `learn.unfreeze()`, when the layer groups (and
   therefore the right lr) have changed.


## Supporting basis and limitations

Built from the fastai v2 source: LRFinder/Learner.lr_find and the valley/slide/minimum/steep suggestion functions in fastai/callback/schedule.py (exponential lr schedule, stop_div at 4x best smoothed loss, num_it capped at batch count, tmp-save weight restoration), Recorder.plot_lr_find, and recurring forums.fast.ai threads on interpreting lr_find plots.

## Change and rationale

New skill: diagnose and fix a broken fastai learn.lr_find() curve.

The LR finder is one of the most-asked-about fastai features on the forums, and most failures (flat curves, immediate divergence, NaN, truncated runs) are misdiagnosed as 'bad learning rate' when they are actually data-pipeline bugs, a silently capped num_it, or stop_div firing. This skill adds a decision procedure that classifies the curve from observable evidence before prescribing a fix, and documents the real v2 mechanics (weight restoration, suggestion trimming, the four suggestion methods).