# Fine-tune a pretrained fastai model without destroying it: freeze/unfreeze schedule and discriminative learning rates

Fine-tune a pretrained fastai model safely: train the head frozen first, then unfreeze with discriminative learning rates via slice(lo, hi); classify post-unfreeze failures (valid_loss spike from too-high body lr, premature unfreeze, overfitting) from the loss curves and apply the matching fix, including gradual freeze_to(-n) unfreezing and SaveModelCallback/EarlyStoppingCallback guards.

Exact reference: {"kind":"skill_version","skill_id":"skl_AfraNLy2XnSiDmpwvvXZrw","version_id":"skv_khBVSnI_xt_P_9ec0QfneQ"}

Applicability: [{"constraint":">=2.0","technology":"fastai","version_scheme":"semver"}]

# Fine-tune a pretrained fastai model without destroying it: freeze/unfreeze schedule and discriminative learning rates

Use this when fine-tuning a pretrained model (`vision_learner`,
`text_classifier_learner`, or any `Learner` with a pretrained backbone)
goes wrong after unfreezing: validation loss spikes, the model overfits
immediately, or unfreezing buys nothing over the frozen head. Classify the
failure from the loss curves first, then apply the matching schedule fix.

## 0. The mechanics you need

```python
learn.freeze()      # freeze_to(-1): train only the last layer group (the head)
learn.unfreeze()    # freeze_to(0):  train every layer group
learn.freeze_to(-2) # freeze all but the last two layer groups (gradual unfreezing)
```

Three facts that cause most of the confusion:

1. **`freeze`/`unfreeze`/`freeze_to` clear the optimizer state.**
   `freeze_to` calls `opt.clear_state()` — momentum and second-moment
   estimates are wiped every time you change what's frozen. Don't try to
   carry optimizer momentum across an unfreeze boundary; plan the schedule
   as separate phases instead.
2. **Discriminative learning rates are a slice, spread log-evenly.**
   `fit_one_cycle(5, slice(1e-6, 1e-3))` gives the earliest layer group
   1e-6, the last group 1e-3, and the middle groups log-spaced values in
   between (`even_mults` over the parameter groups). `slice(1e-3)` alone
   (no start) means `[1e-4, ..., 1e-4, 1e-3]` — every group except the last
   gets `stop/10`. Early groups (generic features) always want the small
   end; the head wants the large end.
3. **`fit_one_cycle`'s lr is the peak, not the starting rate.**
   `fit_one_cycle(n_epoch, lr_max, div=25., div_final=1e5, pct_start=0.25)`
   warms up from `lr_max/25` to `lr_max`, then cosine-anneals down to
   `lr_max/1e5`. So a suggested lr of 3e-3 peaks at 3e-3 mid-training.

The standard recipe (frozen head first, then discriminative unfreeze):

```python
learn.freeze()
lr = learn.lr_find().valley          # suggestion named tuple; .valley/.minimum/.steep/.slide
learn.fit_one_cycle(4, lr)

learn.unfreeze()
lr2 = learn.lr_find().valley         # re-run: different layer groups, different curve
learn.fit_one_cycle(6, slice(lr2/100, lr2))
```

Re-running `lr_find()` after unfreezing is not optional hygiene — the
unfrozen model has different layer groups and a different loss landscape,
so the frozen-phase lr is the wrong answer for the body.

## 1. Classify the failure from the curves

Plot with `learn.recorder.plot_loss()` and compare against the frozen
phase. Note: in fastai v2 the monitored value is `valid_loss` (not the v1
name `val_loss`) — old forum snippets using `val_loss` will silently
monitor nothing.

### A. `valid_loss` spikes immediately after unfreezing and never recovers

The learning rate is too high for the early layer groups — the pretrained
body is being destroyed, not fine-tuned. Observable: the very first
unfrozen epoch is much worse than the best frozen epoch.

- Recover: reload the best frozen weights. If you trained the frozen
  phase with `SaveModelCallback(monitor='valid_loss', fname='head')`,
  `learn.load('head')` gets them back; otherwise retrain the head.
- Redo the unfreeze with a discriminative slice whose low end is 10–100x
  below the frozen lr: `learn.fit_one_cycle(4, slice(lr/100, lr))`.
  Passing a single float after `unfreeze()` applies the *same* lr to every
  group — including the early ones — which is exactly what destroys them.

### B. Frozen-phase `valid_loss` was still decreasing when you unfroze

You unfroze early. An undertrained head plus an unfrozen body mostly buys
overfitting, not accuracy (a recurring finding on the fastai forums: once
the frozen model stops improving, try unfreezing; while it's still
improving, don't).

- Fix: go back to `learn.freeze()` and train the head until `valid_loss`
  plateaus, *then* unfreeze. Judge "plateau" from the recorder values,
  not from a fixed epoch count.

### C. Unfrozen `valid_loss` improves for an epoch or two, then rises while train loss keeps falling

Classic overfitting — the unfrozen model has far more capacity than the
frozen head. Observable: train/valid curves diverge.

- Train fewer unfrozen epochs; the unfrozen phase usually needs fewer
  epochs than the frozen phase.
- Add `EarlyStoppingCallback(monitor='valid_loss', patience=2)` and
  `SaveModelCallback(monitor='valid_loss')` so the best weights survive
  even if you overshoot:

```python
from fastai.callback.tracker import SaveModelCallback, EarlyStoppingCallback
learn.fit_one_cycle(8, slice(lr/100, lr),
                    cbs=[SaveModelCallback(monitor='valid_loss'),
                         EarlyStoppingCallback(monitor='valid_loss', patience=2)])
```

- Increase weight decay (`wd` argument to `fit_one_cycle`) before
  reaching for a smaller model.

### D. Gradual unfreezing: unfreezing one group at a time

When full unfreezing (A) keeps destroying the body even at low lr — common
with small datasets or when the pretrained domain is far from yours —
unfreeze progressively, ULMFiT-style:

```python
learn.freeze()
learn.fit_one_cycle(3, 1e-2)                 # head only
learn.freeze_to(-2)
learn.fit_one_cycle(3, slice(1e-2/2.6**4, 1e-2))  # last two groups
learn.freeze_to(-3)
learn.fit_one_cycle(3, slice(5e-3/2.6**4, 5e-3)) # last three groups
learn.unfreeze()
learn.fit_one_cycle(4, slice(1e-3/2.6**4, 1e-3)) # everything
```

Each `freeze_to` clears optimizer state (fact 1), so treat every phase as
a fresh optimization run with its own lr. Lower the slice's top end as
more groups join — the earlier groups must move less.

## 2. Pre-flight checks before any unfreeze

1. `len(learn.opt.param_lists)` — know how many layer groups the slice
   will spread across. A `vision_learner` resnet34 has 3; a single float
   lr after unfreeze hits all of them equally.
2. Frozen `valid_loss` has plateaued (failure B).
3. `SaveModelCallback(monitor='valid_loss')` is attached so a bad
   unfreeze is recoverable with `learn.load('model')` — the default
   filename is `'model'` in `learn.path/learn.model_dir`.
4. The unfrozen lr comes from an `lr_find()` run *after* `unfreeze()`,
   not from the frozen phase.

## 3. Checklist

1. Always train the head frozen first; unfreeze only on a plateau.
2. After `unfreeze()`, re-run `lr_find()` and use a `slice(lo, hi)`
   discriminative lr — never a single float for all groups.
3. Optimizer state resets on every freeze/unfreeze; schedule in
   independent phases.
4. Guard every unfrozen run with `SaveModelCallback` + early stopping on
   `valid_loss` (v2 name, not `val_loss`).
5. If full unfreezing destroys the body, go gradual with `freeze_to(-n)`.


## Supporting basis and limitations

Built from the fastai v2 source: Learner.freeze/freeze_to/unfreeze and opt.clear_state() in fastai/learner.py, Optimizer.set_hyper slice handling (even_mults log spacing) in fastai/optimizer.py, fit_one_cycle (div=25., div_final=1e5, pct_start=0.25) in fastai/callback/schedule.py, SaveModelCallback/EarlyStoppingCallback (monitor='valid_loss') in fastai/callback/tracker.py, and recurring forums.fast.ai threads on frozen-vs-unfrozen training and overfitting after unfreeze.

## Change and rationale

New skill: freeze/unfreeze fine-tuning schedule and discriminative learning rates for fastai.

Post-unfreeze validation-loss spikes and immediate overfitting are among the most common fastai fine-tuning questions, and the usual failure is applying a single learning rate to all layer groups after unfreeze, or unfreezing before the head converges. This skill adds a decision procedure that classifies the failure from the train/valid curves and documents the real mechanics agents get wrong: freeze_to clears optimizer state, slice(lr) spreads log-evenly across groups, fit_one_cycle's lr is the peak, and v2 monitors valid_loss not val_loss.