Transfer Learning: What to Freeze, What to Fine-Tune, and When It Fails

Transfer learning works because neural networks learn reusable representations. The question is which representations to reuse and which to replace. The answer depends on three variables: source-target similarity, target dataset size, and where in the network the relevant features live. Get this wrong and fine-tuning makes things worse.

Transfer learning is the practice of starting from a model pretrained on a large dataset and adapting it to a new task. It works because early network layers learn general features — edge detectors, colour gradients, texture patterns — that are reusable across tasks. Later layers learn task-specific features that need to be replaced. The question is where to draw the line.

Why representations transfer

A ResNet trained on ImageNet learns a hierarchy of visual features (see Post 52). The first few layers detect oriented edges and blobs — genuinely universal visual primitives that appear in any image-based task. Middle layers detect textures and object parts. Later layers encode ImageNet-specific concepts (dog breeds, car models). For a new task like medical image classification, the early and middle representations are immediately useful; the late representations need to be replaced.

The same logic applies in NLP. BERT's early layers learn syntax and morphology; later layers learn task-specific semantics. Fine-tuning BERT for sentiment analysis can adapt the later layers while preserving early syntactic representations.

The four scenarios: what to do in each

Small dataset, similar domain: freeze all pretrained layers, train only the classification head. The features are directly applicable; retraining with limited data would overwrite them with noise. This is the most common case for industry fine-tuning (e.g., medical imaging on a pretrained ImageNet backbone).

Small dataset, different domain: this is the hardest case. The pretrained features may not be relevant. Options: fine-tune only the last few layers (highest risk of overwriting useful early features), use stronger regularisation and a very low learning rate, or collect more data. There is no reliable recipe.

Large dataset, similar domain: fine-tune the whole network with a low learning rate. The pretrained weights are a good initialisation; you have enough data to adapt all layers carefully.

Large dataset, different domain: fine-tune from scratch, or from pretrained weights with standard learning rates throughout. The pretrained initialisation still helps convergence even if the domain is different.

Learning rate schedules for fine-tuning

A common mistake: applying a uniform learning rate to all layers. Later layers need larger updates (their features are less transferable); earlier layers need very small updates (overwriting general features causes regression). Discriminative fine-tuning (ULMFiT, Howard & Ruder, 2018) uses different learning rates for different layer groups — typically a 10× reduction per group from the output layer toward the input. The output layer gets η, the next group gets η/10, and so on.

Domain adaptation: when the distribution shifts

Fine-tuning assumes the target dataset is representative of the deployment distribution. When it is not — different demographics, different imaging equipment, different writing styles — standard fine-tuning overfits to the fine-tuning distribution. Domain adaptation methods (Domain-Adversarial Neural Networks, adversarial fine-tuning) explicitly learn representations that are invariant to domain, forcing the model to capture task-relevant features rather than distribution-specific artifacts.

LoRA: efficient fine-tuning of large models

Full fine-tuning of a large language model (billions of parameters) requires storing and updating all parameters — computationally expensive and memory-intensive. LoRA (Hu et al., 2022) freezes the pretrained weights and adds low-rank update matrices alongside the original weights: W' = W + ΔW = W + BA, where B ∈ R^{d×r} and A ∈ R^{r×k} with r << min(d,k). Only A and B are trained. With r=8, LoRA reduces trainable parameters by 10,000× for a 7B model while achieving near-full fine-tuning quality. It has become the standard method for adapting LLMs to new tasks or styles.

When transfer learning hurts: negative transfer

Transfer learning can degrade performance if the source and target tasks are negatively correlated — if the pretrained representations actively mislead the model on the target task. This is rare but documented: models pretrained on sentiment-charged text can hurt performance on emotionally neutral classification tasks. The signal is: fine-tuned model performs worse than training from scratch on the target data alone. If this happens, reduce the number of frozen layers or use a lower learning rate for the transferred portions.

Try on Colab: take ResNet-18 pretrained on ImageNet. Fine-tune it on a small medical image dataset (e.g., chest X-ray binary classification, 500 samples). Compare three regimes: (1) train only the final layer, (2) fine-tune the last two blocks + final layer, (3) fine-tune the whole network. Plot validation accuracy vs epoch for all three. Regime 1 should win on small data; regime 3 should overfit.