Immutable solution version
Fine-tune a pretrained fastai model without destroying it: freeze/unfreeze schedule and discriminative learning rates
Complete solution
Fine-tune a pretrained fastai model safely: train the head frozen first, then unfreeze with discriminative learning rates via slice(lo, hi); classify post-unfreeze failures (valid_loss spike from too-high body lr, premature unfreeze, overfitting) from the loss curves and apply the matching fix, including gradual freeze_to(-n) unfreezing and SaveModelCallback/EarlyStoppingCallback guards.
When it applies
[{"constraint": ">=2.0", "technology": "fastai", "version_scheme": "semver"}]
Fine-tune a pretrained fastai model without destroying it: freeze/unfreeze schedule and discriminative learning rates
Use this when fine-tuning a pretrained model (vision_learner, text_classifier_learner, or any Learner with a pretrained backbone) goes wrong after unfreezing: validation loss spikes, the model overfits immediately, or unfreezing buys nothing over the frozen head. Classify the failure from the loss curves first, then apply the matching schedule fix.
0. The mechanics you need
learn.freeze() # freeze_to(-1): train only the last layer group (the head)
learn.unfreeze() # freeze_to(0): train every layer group
learn.freeze_to(-2) # freeze all but the last two layer groups (gradual unfreezing)Three facts that cause most of the confusion:
freeze/unfreeze/freeze_toclear the optimizer state.freeze_tocallsopt.clear_state()— momentum and second-moment estimates are wiped every time you change what's frozen. Don't try to carry optimizer momentum across an unfreeze boundary; plan the schedule as separate phases instead.Discriminative learning rates are a slice, spread log-evenly.
fit_one_cycle(5, slice(1e-6, 1e-3))gives the earliest layer group 1e-6, the last group 1e-3, and the middle groups log-spaced values in between (even_multsover the parameter groups).slice(1e-3)alone (no start) means[1e-4, ..., 1e-4, 1e-3]— every group except the last getsstop/10. Early groups (generic features) always want the small end; the head wants the large end.fit_one_cycle's lr is the peak, not the starting rate.fit_one_cycle(n_epoch, lr_max, div=25., div_final=1e5, pct_start=0.25)warms up fromlr_max/25tolr_max, then cosine-anneals down tolr_max/1e5. So a suggested lr of 3e-3 peaks at 3e-3 mid-training.
The standard recipe (frozen head first, then discriminative unfreeze):
learn.freeze()
lr = learn.lr_find().valley # suggestion named tuple; .valley/.minimum/.steep/.slide
learn.fit_one_cycle(4, lr)
learn.unfreeze()
lr2 = learn.lr_find().valley # re-run: different layer groups, different curve
learn.fit_one_cycle(6, slice(lr2/100, lr2))Re-running lr_find() after unfreezing is not optional hygiene — the unfrozen model has different layer groups and a different loss landscape, so the frozen-phase lr is the wrong answer for the body.
1. Classify the failure from the curves
Plot with learn.recorder.plot_loss() and compare against the frozen phase. Note: in fastai v2 the monitored value is valid_loss (not the v1 name val_loss) — old forum snippets using val_loss will silently monitor nothing.
A. valid_loss spikes immediately after unfreezing and never recovers
The learning rate is too high for the early layer groups — the pretrained body is being destroyed, not fine-tuned. Observable: the very first unfrozen epoch is much worse than the best frozen epoch.
Recover: reload the best frozen weights. If you trained the frozen phase with
SaveModelCallback(monitor='valid_loss', fname='head'),learn.load('head')gets them back; otherwise retrain the head.Redo the unfreeze with a discriminative slice whose low end is 10–100x below the frozen lr:
learn.fit_one_cycle(4, slice(lr/100, lr)). Passing a single float afterunfreeze()applies the same lr to every group — including the early ones — which is exactly what destroys them.
B. Frozen-phase valid_loss was still decreasing when you unfroze
You unfroze early. An undertrained head plus an unfrozen body mostly buys overfitting, not accuracy (a recurring finding on the fastai forums: once the frozen model stops improving, try unfreezing; while it's still improving, don't).
Fix: go back to
learn.freeze()and train the head untilvalid_lossplateaus, then unfreeze. Judge "plateau" from the recorder values, not from a fixed epoch count.
C. Unfrozen valid_loss improves for an epoch or two, then rises while train loss keeps falling
Classic overfitting — the unfrozen model has far more capacity than the frozen head. Observable: train/valid curves diverge.
Train fewer unfrozen epochs; the unfrozen phase usually needs fewer epochs than the frozen phase.
Add
EarlyStoppingCallback(monitor='valid_loss', patience=2)andSaveModelCallback(monitor='valid_loss')so the best weights survive even if you overshoot:
from fastai.callback.tracker import SaveModelCallback, EarlyStoppingCallback
learn.fit_one_cycle(8, slice(lr/100, lr),
cbs=[SaveModelCallback(monitor='valid_loss'),
EarlyStoppingCallback(monitor='valid_loss', patience=2)])Increase weight decay (
wdargument tofit_one_cycle) before reaching for a smaller model.
D. Gradual unfreezing: unfreezing one group at a time
When full unfreezing (A) keeps destroying the body even at low lr — common with small datasets or when the pretrained domain is far from yours — unfreeze progressively, ULMFiT-style:
learn.freeze()
learn.fit_one_cycle(3, 1e-2) # head only
learn.freeze_to(-2)
learn.fit_one_cycle(3, slice(1e-2/2.6**4, 1e-2)) # last two groups
learn.freeze_to(-3)
learn.fit_one_cycle(3, slice(5e-3/2.6**4, 5e-3)) # last three groups
learn.unfreeze()
learn.fit_one_cycle(4, slice(1e-3/2.6**4, 1e-3)) # everythingEach freeze_to clears optimizer state (fact 1), so treat every phase as a fresh optimization run with its own lr. Lower the slice's top end as more groups join — the earlier groups must move less.
2. Pre-flight checks before any unfreeze
len(learn.opt.param_lists)— know how many layer groups the slice will spread across. Avision_learnerresnet34 has 3; a single float lr after unfreeze hits all of them equally.- Frozen
valid_losshas plateaued (failure B). SaveModelCallback(monitor='valid_loss')is attached so a bad unfreeze is recoverable withlearn.load('model')— the default filename is'model'inlearn.path/learn.model_dir.The unfrozen lr comes from an
lr_find()run afterunfreeze(), not from the frozen phase.
3. Checklist
- Always train the head frozen first; unfreeze only on a plateau.
After
unfreeze(), re-runlr_find()and use aslice(lo, hi)discriminative lr — never a single float for all groups.Optimizer state resets on every freeze/unfreeze; schedule in independent phases.
Guard every unfrozen run with
SaveModelCallback+ early stopping onvalid_loss(v2 name, notval_loss).- If full unfreezing destroys the body, go gradual with
freeze_to(-n).
Evidence
Built from the fastai v2 source: Learner.freeze/freezeto/unfreeze and opt.clearstate() in fastai/learner.py, Optimizer.sethyper slice handling (evenmults log spacing) in fastai/optimizer.py, fitonecycle (div=25., divfinal=1e5, pctstart=0.25) in fastai/callback/schedule.py, SaveModelCallback/EarlyStoppingCallback (monitor='valid_loss') in fastai/callback/tracker.py, and recurring forums.fast.ai threads on frozen-vs-unfrozen training and overfitting after unfreeze.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_GLoNmxZ52H1XNhx7x0qAbg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Fine-tune a pretrained fastai model without destroying it: freeze/unfreeze schedule and discriminative learning rates <span>— Vectle</span></a>