Pre-training & Transfer Learning

Masked LM, causal LM, BERT vs GPT objectives, feature extraction vs fine-tuning

The Transformer module closed on architecture — how attention, residuals, and Pre-LN normalisation combine into a stable, stackable block. None of that explains how the *weights inside* that architecture come to know anything about language before you've trained them on your task at all. That's a separate question, and it's the one this module answers.

You have 500 labelled radiology reports and need to classify them by findings. Train a model from scratch on those 500, and it has to learn *everything* at once — what "pulmonary" means, that "nodule" is worrying, that "no evidence of" flips the meaning, *and* the actual classification rule — all from 500 examples. It ends up memorising quirks that do not generalise, and test AUC lands at a dismal 0.61.

Now do one thing differently: start from PubMedBERT, a model already trained on 14 million medical papers, and fine-tune it on the *same* 500 reports. Test AUC: 0.87. Nothing about your labels or task changed. What changed is the *starting point* — PubMedBERT already knows medical language, so your 500 labels only have to teach it the final decision, not the entire vocabulary. This is transfer learning, and it is one of the highest-leverage ideas in modern ML.

What pre-training actually does

Picture the model's millions of weights as coordinates on a vast, foggy mountain range, where height measures how badly the model performs at language and low valleys are where it performs well. Training from scratch is being dropped at a *random* point in that fog with only a few hundred noisy update steps — as many as 500 labelled examples can drive — to feel your way toward a good valley; that's barely enough to get off the plateau you happened to land on, which is exactly why the from-scratch radiology model stalled at 0.61.

Pre-training is a helicopter ride to a known region of that range before you ever start hiking. You take a mountain of unlabelled text and make the model play fill-in-the-blank or predict-the-next-word, billions of times — this is *self-supervised* learning, since the "labels" (the missing or next word) come free from the text itself. To get good at that game the model is forced to internalise how language works — grammar, vocabulary, which words go together, domain structure — and all of that gets baked into its weights as coordinates, landing it at a "base camp" a short hike from a huge number of good valleys, including yours. Fine-tuning is that short hike: your 500 examples only have to nudge the model from base camp to the *particular* valley your task needs, not search the whole foggy range from a random drop point.

Two pre-training styles, two strengths

The *game* you make the model play shapes what it becomes. Masked language modelling (BERT) hides about 15% of the words and asks the model to fill them in using context on *both* sides — which produces rich, full-context representations, ideal for *understanding* tasks. Causal language modelling (GPT) predicts each next word from only the words *before* it — which is denser training (every token is a target) and lines up naturally with *generating* text. That is why understanding tasks lean on BERT-style models and generation leans on GPT-style ones.

The one danger: forgetting what it knew

Fine-tuning has a trap called catastrophic forgetting: hit the pre-trained model with a big learning rate on your small dataset and you *overwrite* the very knowledge that made it valuable, collapsing it onto your narrow task. The standard safety recipe is a *gentle* touch — a learning rate 10–100× smaller than pre-training used, only a few epochs, a short warmup, and a little weight decay — small enough steps that you *stay near* the pre-trained starting point instead of wandering off and erasing it.

Key points

• For any NLP or vision task with fewer than 10K labeled examples, start from a pretrained model — the few-shot gains are 20–50 AUC points in specialized domains. The 500-example radiology case is not unusual. Medical NLP, legal document classification, scientific literature tagging — all have small labeled datasets and large unlabeled corpora. Domain-matched pretraining (PubMedBERT for biomedical, LegalBERT for contracts) outperforms general BERT by an additional 5–15 points when the domain vocabulary diverges significantly from web text.
• Trap: catastrophic forgetting — fine-tuning with a high learning rate on a small dataset overwrites pretrained representations. Use a warmup schedule, a learning rate 10–100× smaller than the original pretraining LR, and weight decay. BERT's original pretraining used LR = 1e-4. Fine-tuning at that same rate for 5 epochs on 500 examples destroys the pretrained representations: the model converges to a local minimum defined entirely by the narrow task distribution. The standard recipe: LR = 2e-5, 3–5 epochs, linear warmup over the first 10% of steps, weight decay 0.01. This keeps the parameter updates small enough that the pretrained basin is not abandoned.
• Diagnostic: if validation loss diverges immediately after fine-tuning starts, the LR is too high. If it never improves past random baseline, the task head is misspecified or the pretrained model is mismatched to the domain. Two distinct failure modes look superficially similar (poor validation accuracy) but have opposite causes. Immediate divergence: the first gradient step is too large, destroying the pretrained initialization — reduce LR by 10×. No improvement at all: the pretrained model's representation space does not have a basis for the target task — the domain gap is too large (e.g., using a general English BERT for Chinese clinical text), or the output head projects to the wrong dimension or uses the wrong activation for the task type.

Recap

• Picks up from Transformers: that module covered architecture (attention + residuals + Pre-LN); this one covers what makes the *weights* inside that architecture already know language before your task even starts.
• Transfer learning payoff: 500 radiology reports trained from scratch reach AUC 0.61; the *same* 500 examples fine-tuning PubMedBERT reach 0.87 — a 26-point gain where only the *starting point* changed, not the data. No amount of regularisation from random init replicates it.
• Mountain-range metaphor: weights = coordinates, height = how badly the model performs. From-scratch training is a random drop in the fog with only 500 tries to find a good valley (why it stalled at 0.61). Pre-training is a helicopter ride to a "base camp" already near many good valleys.
• Pre-training is self-supervised: the model does fill-in-the-blank (masked) or next-word prediction on a mountain of *unlabelled* text — labels come free from the text itself — which forces it to internalise grammar, facts, and structure into its weight-coordinates before it ever sees your task.
• Fine-tuning is the short hike from base camp, not a fresh search from a random drop — it moves the weights only a little toward the one valley your task needs, which is why a few hundred labelled examples suffice.
• Two pre-training styles: masked LM (BERT — sees both sides of a token, best for *understanding* tasks) and causal LM (GPT — predicts the next word, gives denser signal since every token is a target, best for *generation*).
• Trap — catastrophic forgetting: a large learning rate on a small dataset overwrites the very pre-trained knowledge that made the model valuable, dragging it back toward random. The small dataset can't re-teach what a big corpus taught.
• Safety recipe: use a learning rate 10–100× smaller than pre-training (~2e-5), few epochs, a short warmup, and weight decay ~0.01 — all to keep the weights *near* the pre-trained basin rather than wandering out of it.
• Diagnostic: immediate divergence in the first steps → LR too high, divide by 10; a model that never beats the from-scratch baseline → domain mismatch or a wrong task head, not a tuning issue.

Check your understanding