Loss Functions: Why You Are Minimising What You Are Minimising

MSE, cross-entropy, KL divergence — every loss function is a specific statistical assumption about the data-generating process. Minimising MSE is equivalent to maximum likelihood estimation under Gaussian noise. Minimising cross-entropy is equivalent to minimising KL divergence between the data distribution and the model. Knowing this makes loss function choice principled rather than arbitrary.

Loss functions are not handed down by convention. Each one encodes a specific belief about how errors should be penalised, which in turn encodes an assumption about the noise model for the data. Choosing the right loss is the same as choosing the right statistical model for your problem.

MSE from the Gaussian likelihood

Suppose you observe data points y_i = f(x_i) + ε_i where ε_i ~ N(0, σ^2). Maximum likelihood estimation asks: what parameters θ maximise the probability of the observed data? The log-likelihood is: log p(y | x, θ) = -1/(2σ^2) Σ (y_i - f(x_i; θ))^2 + constant. Maximising this is identical to minimising Σ (y_i - f(x_i; θ))^2 — mean squared error.

MSE is the right loss when errors are Gaussian and symmetric. The quadratic penalty means large errors are punished much more than small ones. If your residuals have heavy tails (large outliers are common), MSE overfits to those outliers. Mean Absolute Error (MAE) corresponds to a Laplace noise model and is more robust to outliers because the penalty grows linearly.

Cross-entropy from KL divergence

For classification, let p be the true label distribution (one-hot for hard labels) and q be the model's output distribution (after softmax). Cross-entropy: H(p, q) = -Σ p(y) log q(y). For a single correct class c: H(p, q) = -log q(c). This is the negative log probability the model assigns to the correct class — minimising it maximises the probability assigned to correct labels.

The deeper connection: KL(p || q) = H(p, q) - H(p). H(p) is fixed (the entropy of the data distribution), so minimising KL(p || q) is identical to minimising cross-entropy. Training with cross-entropy is performing maximum likelihood estimation via minimising the divergence between the data distribution and the model's distribution.

Why cross-entropy works better than MSE for classification

MSE penalises predicted probabilities quadratically: if the model predicts 0.9 when the true class is 1, MSE loss is 0.01. If it predicts 0.1, loss is 0.81. The gradient at 0.9 is 0.1 — small, encouraging slow updates even when the prediction is clearly wrong from a log-likelihood perspective. Cross-entropy: at prediction 0.9, loss = -log(0.9) = 0.105. At prediction 0.1, loss = -log(0.1) = 2.3. The gradient is larger when predictions are more wrong, regardless of the probability threshold — more informative throughout training.

KL divergence: why it is asymmetric

KL(p || q) ≠ KL(q || p). KL(p || q) penalises regions where p is large but q is small — the model fails to cover modes that the data has. KL(q || p) penalises regions where q is large but p is small — the model assigns probability to regions the data does not support. VAEs minimise KL(q || p) (see Post 62). Reinforcement learning from human feedback (RLHF) adds a KL penalty to prevent the policy from diverging too far from the reference model.

Focal loss: solving class imbalance

For class-imbalanced problems (fraud detection, object detection with many background anchors), easy negatives dominate the loss. The model trains mostly on confident correct predictions that contribute almost no gradient. Focal loss (Lin et al., 2017): FL(p_t) = -(1 - p_t)^γ * log(p_t). The modulating factor (1-p_t)^γ down-weights easy examples (high p_t) and focuses learning on hard ones. γ=2 is the standard setting. Focal loss is the default for single-stage object detectors (RetinaNet, FCOS) and useful whenever training is dominated by easy negatives.

Contrastive and triplet loss: learning metric spaces

For embedding-based models (face recognition, recommendation, semantic search), the goal is not to predict a class but to learn an embedding space where similar items are close and dissimilar items are far. Triplet loss: L(a, p, n) = max(0, d(a, p) - d(a, n) + margin). An anchor a, a positive example p (same class), a negative n (different class). The loss pushes the positive closer and the negative farther than the margin. Hard negative mining — selecting the negatives the model currently rates most similar to the anchor — is critical for training efficiency.

Try on Colab: train a binary classifier with MSE loss and cross-entropy loss on the same dataset. Plot the gradient magnitudes at the output layer across training epochs for both. The cross-entropy gradients will be larger and more informative early in training. Then implement focal loss from scratch and compare it to cross-entropy on an imbalanced dataset (oversample the minority class to 1% of data).