Knowledge Distillation: Why a Small Model Can Learn More from a Big Model Than from Data

A small model trained on soft teacher probabilities learns more than the same model trained on hard labels. This is counterintuitive — the teacher is not giving the student new information it could not access from the original data. But soft probabilities carry a richer signal: they encode the teacher's uncertainty and the relationships between classes. Hinton's 2015 paper made this precise and spawned an entire subfield of model compression.

A large neural network trained to high accuracy encodes more than the final class predictions. Its output probabilities — even for wrong classes — carry information about how similar the classes are. A cat image might get 0.75 probability for "cat," 0.20 for "tiger," 0.04 for "leopard," and 0.001 for "truck." The distribution over wrong classes reveals that the model has learned that cats are more similar to tigers than to trucks. A small student model trained on these soft labels learns this relational structure — information that is simply absent from the hard label "cat=1, everything else=0."

The knowledge distillation objective

Hinton et al. (2015) proposed training a student model on a combination of the hard labels and the teacher's soft predictions: L = α * L_hard(y, σ(z_s)) + (1-α) * L_soft(σ(z_t/T), σ(z_s/T)). σ is softmax, z_s and z_t are student and teacher logits, T is temperature, and α balances the two terms. The soft loss uses a higher temperature T to soften both the teacher and student distributions — making the small probabilities on wrong classes more meaningful. At inference, temperature is set back to 1.

Why soft targets help: the dark knowledge explanation

The teacher's soft predictions encode what Hinton called "dark knowledge" — information about the similarity structure between classes that is not present in hard labels. Consider two student models: one trained on 60,000 MNIST hard labels, one trained on those same labels plus the teacher's soft probabilities. The soft-label model typically outperforms the hard-label model even though both have access to the same images and ground-truth labels. The difference is the dark knowledge: the teacher has learned that 4s look like 9s, that 1s can look like 7s, that 3s sometimes look like 8s. This learned similarity structure speeds up the student's learning.

Intermediate layer matching: deeper distillation

Beyond output probabilities, the student can learn from the teacher's internal representations. FitNets (Romero et al., 2015) trains the student to match the intermediate feature maps of the teacher, not just the final outputs. Attention transfer (Zagoruyko & Komodakis, 2017) matches attention maps — where in the image the teacher focuses. These methods transfer more of the teacher's learned representation and typically outperform output-only distillation.

Data-free distillation

Standard distillation requires the training data to query the teacher. For proprietary models or when training data is unavailable, data-free distillation generates synthetic inputs that maximise the teacher's response diversity (similar to Deep Inversion / dream training). The student trains on these generated examples with the teacher's soft labels. Performance is lower than data-based distillation but enables compression without data access.

Self-distillation and born-again networks

You can distil a model into a copy of itself. Born-again networks (BAN) train a student with the same architecture as the teacher, using the teacher's soft outputs. Surprisingly, the student matches or exceeds the teacher. This is explained by the soft targets reducing overfitting: the student sees a smoother loss surface than the hard-label teacher and finds a better generalising minimum. Repeating this process (BAN1 → BAN2 → BAN3) continues to improve performance for several generations.

Distillation in practice: LLMs

Most deployed LLMs are distilled from larger models. OpenAI distilled GPT-3.5 (text-davinci-003) into smaller, faster models. Google's PaLM 2 variants (Gecko, Otter, Bison) are distilled from the full model. Alpaca fine-tuned LLaMA-7B on outputs from GPT-3.5 (text-davinci-003), achieving instruction-following comparable to the much larger teacher. This SOTA-distillation-from-API approach is now standard in the open-source LLM community: generate diverse (prompt, response) pairs from a capable teacher, fine-tune a smaller student on them. The student learns the teacher's style and instruction-following behaviour without the teacher's parameter count.

Try on Colab: train ResNet-110 (teacher) on CIFAR-10 to ~93% accuracy. Distil it into ResNet-20 (student) using: (a) hard labels only, (b) soft teacher labels (T=4, α=0.1). Compare final test accuracy for both student variants. The distilled student should outperform the non-distilled student by 0.5-1.5% despite seeing the same training data.