Advanced 29 min read fine-tuningLoRAPEFTadaptersLLM

Fine-Tuning Strategies

Full fine-tune, LoRA, prefix tuning, adapter layers — when each applies

Pre-training's own closing flagged the risk: fine-tune too aggressively and you overwrite the knowledge that made the model valuable. This module is about how to make the correction small enough that the risk mostly disappears. You want LLaMA-2 70B to answer internal questions in your company's tone and format. The brute-force way — full fine-tuning — means updating *all 70 billion* weights. Just the bookkeeping (16-bit weights and gradients, plus Adam's two 32-bit running averages) needs roughly 12 bytes per parameter — about 840 GB of GPU memory, a dozen top-end GPUs, days of training, and a real risk of overwriting the very abilities that made the model good. For most teams that is a non-starter.

But look at what actually has to change. Teaching a 70-billion-parameter model your company's tone doesn't require touching what it already knows about grammar, facts, or reasoning — it requires a small, structured *correction* layered on top of what's already there. Picture a sticky note stuck on one page of a huge textbook: you don't retype the page, you leave a small note in the margin, and reading proceeds normally except wherever the note applies. If the correction a task genuinely needs is small and structured, the note can be tiny compared to the page it corrects.

Making that concrete

Take a representative 4096×4096 weight matrix — a typical attention-projection size at this scale — 16.7 million numbers. A full update to it would itself be a 4096×4096 matrix: 16.7 million more numbers to learn from your comparatively tiny fine-tuning dataset. But suppose the *correction* that matrix needs can be described by a much narrower structure — the product of two skinny matrices, one 4096×8 and one 8×4096. Multiply them out and you get a full 4096×4096 update, but you only had to *learn* the two skinny factors: 4096×8 + 8×4096 = 65,536 numbers — a 99.6% cut from 16.7 million.

That is the whole trick behind LoRA (Low-Rank Adaptation): freeze the giant base model exactly as it is, and train only the two skinny "sticky-note" matrices, B and A. Once trained, the tiny update merges *straight back* into the original weights — new W = W + B·A — so the deployed model is exactly the same size and speed as the original: zero extra inference cost. That mergeability is LoRA's edge over "adapter" approaches that bolt on permanent extra modules the model must run through forever.

The next crisis: the frozen base is still huge

LoRA already cuts *trainable* parameters by 99.6%. But the frozen base model still has to sit in GPU memory at full size — 70B parameters at 16-bit precision is still about 140GB, more than any single GPU holds. Can the frozen part be shrunk too, without hurting the tiny trainable part's accuracy?

Squash the frozen base down to 4-bit numbers (from ~140GB to ~35GB), and train the small LoRA matrices in full precision on top. Because the base weights are never actually updated — all the learning happens in the little adapters — the rounding error from the 4-bit squashing barely matters to the final result. This combination — a 4-bit frozen base plus full-precision LoRA matrices — is what's called QLoRA, and it is what lets a 70B fine-tune fit on a single 80GB GPU, at near-full-fine-tuning quality.

One distinction worth nailing down

People sometimes call "train a fresh classifier on top of a frozen model" *fine-tuning* — it is not, it is feature extraction, and it can only *re-sort* the representations the model already has. True fine-tuning (including LoRA) actually changes the model's effective weights, which is what lets it learn genuinely new behaviour — a new tone, a new format, a new skill. If you need the model to *do* something new, you need real fine-tuning; if you only need to classify what it already understands, feature extraction is enough.

Key points

Start with LoRA for any fine-tuning task — it matches full fine-tuning quality in most cases with 10–100× less compute and no catastrophic forgetting risk. Use rank 8–16 for most tasks, rank 64+ only if you need to teach genuinely new knowledge. For teaching a new response format, tone, or domain-specific behavior that the pretrained model already has partial knowledge of, rank 8–16 captures the adaptation signal. For teaching a model new facts it was not exposed to in pretraining — new entities, new languages, new structured formats — higher rank or full fine-tuning is necessary. Start low and increase rank only if validation performance plateaus.
Trap: fine-tuning on fewer than 1K examples with a low learning rate often gives no improvement over the base model — the signal is too weak. Either collect more data, use a stronger learning rate with warmup, or use prompt engineering instead. LoRA with rank 8 has ~65K trainable parameters for a single 4096×4096 matrix. With 500 training examples, the signal-to-noise ratio for those parameters is marginal. If validation performance is flat after 3 epochs, the problem is data volume, not architecture. LoRA at rank 16 on 500 examples is not materially different from prompt engineering — both are adapting a powerful pretrained model with very thin signal.
Diagnostic: monitor training loss AND a held-out prompt that captures the target behavior. If training loss decreases but the target behavior does not improve, the fine-tuning data does not actually demonstrate the target behavior. This is the most common fine-tuning failure: the training data measures something correlated with but not identical to what you want. If you want the model to always respond in bullet points, your training data must contain bullet-point responses — training loss on paragraph responses will decrease without producing the format change. Log the target-behavior metric (format compliance, accuracy on held-out examples, human evaluation scores) separately from training loss throughout the run.

Takeaway

Task-specific weight updates have low intrinsic rank — LoRA exploits this to fine-tune a 70B model while training only a small fraction of its total parameters (typically well under 1% across the whole model; a single 4096×4096 matrix's own update is a 99.6% cut, from 16.7M to 65K trainable numbers), and because BA merges directly into W after training, the deployed model is byte-for-byte identical to the base model with zero inference overhead.

Recap

Full fine-tuning is brutal: all 70B weights → ~14 bytes/param ≈ 980GB, a dozen GPUs, and risk of overwriting what made the model good.
LoRA insight: the needed update is low-rank. Learn a 4096×4096 update as two skinny matrices (4096×8, 8×4096) — ~65K numbers, a 99.6% cut. Base model frozen.
LoRA merges back into W: deployed model is same size and speed as the original — zero extra inference cost (its edge over adapters).
QLoRA: squash frozen base to 4-bit (~140GB → ~35GB), train full-precision LoRA on top → 70B fine-tune on one 80GB GPU; rounding error barely matters since base weights never update.
Rank guidance: 8–16 for tone/format/behaviour the model partly knows; 64+ or full fine-tune for genuinely new knowledge.
Feature extraction ≠ fine-tuning: a fresh classifier on a frozen model only re-sorts existing representations; real fine-tuning (incl. LoRA) changes effective weights to learn new behaviour.
Diagnostic: if training loss drops but target behaviour doesn't, the data doesn't actually demonstrate the target behaviour.

Check your understanding

Q1. You fine-tune a pre-trained BERT model on a sentiment classification task with 500 labeled examples. What are the risks, and what techniques do you use? Select the TWO correct statements.

A) Risks: catastrophic forgetting from large updates to all 110M weights on a small task, and overfitting since 500 examples is far too few to safely update every layer. Techniques: freeze early layers, use a small LR (1e-5/2e-5) on pre-trained layers, few epochs with early stopping.
B) Adapters or LoRA on top of a frozen BERT (~0.1% trainable parameters) plus data augmentation (back-translation, synonym replacement to expand 500→~2000 effective examples) sharply reduce both forgetting and overfitting risk versus full fine-tuning.
C) The primary risk is underfitting, since BERT needs at least 10,000 examples per class to move away from general language modelling; the fix is a much higher learning rate (1e-3) to force aggressive adaptation from the limited data.
D) With 220,000 parameters per example, the ratio actually favors full fine-tuning rather than risking it, since BERT's pre-trained representations prevent the model from learning noise; the main risk left is underfitting from too small a learning rate.

Q2. LoRA (Low-Rank Adaptation) decomposes weight updates into low-rank matrices: ΔW = BA where B ∈ ℝ^{d×r} and A ∈ ℝ^{r×k}. For d=k=1024 and r=8, how many parameters does LoRA add vs full fine-tuning? Why does this work?

A) LoRA adds 16,384 params (8,192 each for B and A), but still needs gradient tracking through all frozen base weights to compute A/B's gradients — so training memory matches full fine-tuning, with savings appearing only at inference once BA is merged.
B) LoRA's savings come from targeting only attention matrices (Q,K,V,O), not from low-rank factorisation itself: 12×4×16,384=786,432 updated parameters vs 12×4×1,048,576 for full attention fine-tuning, while FFN layers still get full-rank updates.
C) Full fine-tuning needs 1,048,576 params per matrix; LoRA needs 8,192(B)+8,192(A)=16,384 — a 64× cut. This works because adaptations empirically have low intrinsic dimensionality, and at inference W'=W+BA merges with zero added latency.
D) LoRA needs 16,384 params, a 64× cut, but not because of low intrinsic rank — BA is always full-rank with independent random init, so rank-8 can represent any full-rank update. The real benefit is implicit regularisation equivalent to nuclear-norm minimisation.

Q3. When does RLHF (Reinforcement Learning from Human Feedback) improve a language model beyond standard fine-tuning? What is the reward model, and what can go wrong?

A) RLHF helps mainly on verifiable tasks (code execution, math) where an automated checker replaces the reward model and PPO can explore beyond the SFT dataset; on subjective tasks RLHF underperforms SFT since preference data is too noisy for reliable rewards.
B) RLHF helps when "helpful" can't be reduced to cross-entropy. A reward model trained on human comparisons scores responses, PPO optimises against it. Failure modes: reward hacking, sycophancy, collapsed diversity.
C) RLHF only helps once a model exceeds 7B parameters, since smaller models' reward models — typically logistic regression on frozen embeddings — can't learn preferences accurately enough for PPO to beat the SFT baseline.
D) RLHF and SFT are equivalent in expectation, both fitting human-preferred outputs from the same data; RLHF's only real advantage is that PPO needs 10× less annotation, and its failure mode is purely PPO's clipping parameter causing policy divergence.

Try it interactively

ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.

Open ML Systems Lab →