Logistic Regression From Scratch: MLE, the GLM Connection, and Why It Still Matters
Logistic regression is the model every ML interview assumes you know cold — not sklearn.LogisticRegression() but the actual mathematics. Where does the sigmoid come from? Why is log-loss the right loss? What does the weight on a feature actually mean? This post derives it all from first principles and covers the GLM framework it sits inside.
Logistic regression is the canonical binary classification model. It is used in production at every major tech company for click-through rate prediction, churn modelling, and credit risk scoring. Its simplicity makes it interpretable; its calibration properties make it reliable. But most ML practitioners cannot derive it from first principles. This post fixes that.
The modelling choice: sigmoid
For binary classification (y ∈ {0, 1}), we want to model P(y=1|x). This probability must lie in [0,1], but a linear model wᵀx can output any real number. We need a link function that squashes ℝ to (0,1). The sigmoid: σ(z) = 1/(1+e^(-z)) = e^z/(1+e^z). It is S-shaped, differentiable everywhere, σ(0)=0.5, σ(z)→1 as z→∞, σ(z)→0 as z→-∞. The logistic regression model: P(y=1|x,w) = σ(wᵀx + b). But why sigmoid specifically, and not tanh or any other S-shaped function?
The GLM derivation: sigmoid falls out of the exponential family
Generalised Linear Models (GLMs) answer "which link function is correct for which distribution." A GLM has three components: random component (the distribution of y), systematic component (the linear predictor wᵀx), and link function connecting them. The Bernoulli distribution is in the exponential family: P(y|η) = exp(ηy - log(1+e^η)) where η is the natural parameter. If we set η = wᵀx (the linear predictor equals the natural parameter — the canonical link), then: P(y=1|x,w) = e^(wᵀx) / (1 + e^(wᵀx)) = σ(wᵀx). The sigmoid falls out of the Bernoulli exponential family with the canonical link. It is not an arbitrary choice — it is the mathematically natural choice.
Log-odds interpretation
log(P(y=1)/P(y=0)) = log(σ(wᵀx)/(1-σ(wᵀx))) = wᵀx. The log-odds (logit) is linear in the features. This is where "logistic" comes from (logit = log-odds). Each weight wⱼ is the change in log-odds per unit change in feature xⱼ, holding all other features constant. Exponentiating: e^wⱼ is the odds ratio for feature j. If wⱼ = 0.7, then e^0.7 ≈ 2 — a one-unit increase in xⱼ approximately doubles the odds of y=1.
Deriving the loss function from MLE
Label model: P(y|x,w) = σ(wᵀx)^y (1-σ(wᵀx))^(1-y). Log-likelihood: ℓ(w) = Σᵢ [yᵢ log σ(wᵀxᵢ) + (1-yᵢ) log(1-σ(wᵀxᵢ))]. Negate to get the loss (binary cross-entropy): L(w) = -Σᵢ [yᵢ log σ(wᵀxᵢ) + (1-yᵢ) log(1-σ(wᵀxᵢ))]. Gradient: ∂L/∂w = Σᵢ (σ(wᵀxᵢ) - yᵢ) xᵢ = Xᵀ(ŷ - y). This is the residual (prediction minus label) times the feature vector — exactly the same form as the linear regression gradient. This is not coincidence: GLMs all have the same gradient structure.
Training: gradient descent and its variants
There is no closed-form solution for logistic regression weights (unlike linear regression). We use iterative optimisation. Full gradient descent: w ← w - α Xᵀ(ŷ - y). Mini-batch SGD: sample a batch, compute gradient on batch, update. Newton's method converges in fewer iterations: w ← w - H⁻¹ ∇L where H = Xᵀ W X (W = diag(σ(1-σ)) is the weight matrix). Newton is expensive (O(d²) per step) but used in the IRLS (Iteratively Reweighted Least Squares) implementation.
Regularisation
L2 (Ridge): L_reg = L + (λ/2)||w||². Gradient adds λw. Shrinks all weights toward zero, never to exactly zero. Equivalent to Gaussian prior on weights. L1 (Lasso): L_reg = L + λ||w||₁. Produces sparse solutions — some weights exactly zero (feature selection). Equivalent to Laplace prior. Elastic Net: L + λ₁||w||₁ + λ₂||w||². Combines sparsity and shrinkage.
Multiclass: Softmax regression
For K > 2 classes, logistic regression generalises to softmax regression (also called multinomial logistic regression): P(y=k|x,W) = exp(wₖᵀx) / Σⱼ exp(wⱼᵀx). One weight vector per class. The loss is categorical cross-entropy: -Σᵢ Σₖ y_{ik} log P(y=k|xᵢ,W). Softmax regression is the output layer of every neural network classification head.
Interview questions on this topic
"Why does logistic regression output probabilities rather than just 0/1 labels?" — Because the model is P(y=1|x) under the Bernoulli distribution. The sigmoid maps the real-valued linear score to a valid probability. This allows uncertainty quantification and calibration.
"What is the decision boundary of logistic regression? What shape is it?" — The decision boundary is where P(y=1|x) = 0.5, i.e., σ(wᵀx) = 0.5, i.e., wᵀx = 0. This is a hyperplane — logistic regression is a linear classifier. It can separate linearly separable data only. For non-linear boundaries, use feature engineering (polynomial features) or a nonlinear model.
"Why does logistic regression with L2 regularisation never have exactly zero weights, but L1 does?" — L2 penalty λw_j²: gradient is λw_j, which is smooth and approaches zero as w_j→0 but never becomes exactly zero. L1 penalty λ|w_j|: the subgradient at w_j=0 is the interval [-λ,λ], which includes zero — if the loss gradient is small enough, the optimizer can stay at w_j=0.
"You have a 1:100 positive-to-negative class imbalance. How does this affect logistic regression training and inference?" — The model is pushed toward predicting the majority class. Solutions: reweight the loss by class frequency (positive class weight = 100x), oversample positives (SMOTE), undersample negatives, or lower the classification threshold below 0.5 at inference time. The log-likelihood gradient is dominated by the majority class without reweighting.
Try on Colab: implement logistic regression from scratch using only NumPy. Use gradient descent with learning rate schedule and L2 regularisation. Train on the Breast Cancer Wisconsin dataset. Plot the training loss curve and verify it matches sklearn.LogisticRegression with the same regularisation. Then implement Newton's method (IRLS) and compare convergence speed (iterations to reach 1e-6 gradient norm).