Logistic Regression

Sigmoid, cross-entropy loss, decision boundary, calibration

The linear regression module ended by picking the right yardstick for predicting a *number* — MAE, RMSE, R². But not every prediction is a number. Here is a question doctors have asked for a very long time: will this patient have a heart attack in the next ten years? You cannot answer that honestly with a flat yes or no — nobody knows the future. What you *can* give is a probability: this patient has a 12% chance. That is the real job. It is a classification problem — the outcome is one of two classes, heart attack or not — but we do not want a bare label, we want a number between 0 and 1 we can trust. Logistic regression is the tool that has quietly done this job for medicine, banking, and half the internet for decades. Let me show you how it pulls it off.

There is a nice bit of history hiding in the name. Almost two hundred years ago a mathematician named Verhulst was studying how populations grow — not in a straight line, but slow at first, then fast, then flattening out as food and space run low. He drew that S-shaped curve and called it the logistic curve. Decades later people noticed the very same S-curve is perfect for a completely different task: taking any number and gently squashing it into a probability between 0 and 1. That borrowed curve is the engine we are about to build.

The setup

Start with what we already know how to build: a plain linear equation, w·x + b. Feed in the patient's numbers — age, blood pressure, cholesterol — and out comes a single number. But here is the snag. That number lives on the whole number line: it could be −4, or 3000. A linear equation will happily hand you 1.4 or −0.3, and those are nonsense as probabilities. So the one question that *defines* logistic regression is this: how do we bend the wide-open output of a linear equation down into the (0, 1) range of a probability?

Building the bridge

The trick uses a pair of functions that undo each other. You already know one such pair: eˣ and its inverse, the natural log. eˣ takes any number and gives back a positive one — its output lives in (0, ∞). The natural log runs it backwards: hand it a positive number, it gives back any number at all. From that pair we build a second pair — the logit and the sigmoid — which also undo each other. The logit takes a probability in (0, 1) and stretches it out across the whole number line. The sigmoid, $σ(z) = 1/(1 + e^{-z})$, does the reverse: it takes any number and squashes it into (0, 1). That squash is exactly the bend we were hunting for — and yes, the sigmoid is the same S-curve Verhulst drew.

Now the move that makes everything click. A linear equation outputs a number on the whole line. A logit is also a number on the whole line. So instead of forcing the linear equation to spit out a probability directly, we let it predict the logit, then run that through the sigmoid to land on a clean probability. The linear part does what it is naturally good at; the sigmoid handles the bending.

Make this concrete with numbers you can check by hand. Suppose the model has already been fit — trained by minimizing the loss we're about to define, via gradient descent, the same way any model learns its weights — and it has settled on a single feature x (a patient's cholesterol level, standardized so 0 is average), with weight w = 1.4 and bias b = −0.2. Patient 1 has x = 0.5. The linear part gives the logit: z = w·x + b = 1.4×0.5 − 0.2 = 0.7 − 0.2 = 0.5. Run that through the sigmoid: σ(0.5) = 1/(1+e⁻⁰·⁵) = 1/(1+0.6065) = 1/1.6065 ≈ 0.622. So this patient's predicted probability of a heart attack is about 62.2%. Patient 2 has x = −1.0: z = 1.4×(−1) − 0.2 = −1.6, and σ(−1.6) = 1/(1+e¹·⁶) = 1/(1+4.953) ≈ 0.168 — a 16.8% predicted risk. Same weights, same formula, two very different numbers — because the logit moved from 0.5 to −1.6.

But what is a logit, really?

Here is the part most courses rush past, and it is the heart of the whole thing. A logit is the log of the odds.

Odds are just a way of comparing the two outcomes: the chance of the event divided by the chance of no event. Take Patient 1's own probability, 0.622: the odds are 0.622 / (1−0.622) = 0.622/0.378 ≈ 1.65 — "about 1.65 to 1." Odds have an annoying lopsidedness, though. A probability of 0.99 gives odds of 99. Its mirror image, a probability of 0.01, gives odds of 0.01. Same distance from the middle, yet one number is 99 and the other a tiny sliver — you cannot line them up on a fair scale.

Wrapping the odds in a log fixes the lopsidedness at once. log(99) ≈ +4.6 and log(0.01) ≈ −4.6 — now they are clean mirror images around zero. That log-of-odds is the logit, and it is exactly the quantity our linear equation predicts: check it against Patient 1 — log(1.65) ≈ 0.5, matching the z = 0.5 computed above exactly, because that's what "logit" means. So the full pipeline is: linear equation → logit (log-odds) → sigmoid → probability.

And this hands us something lovely: one weight, read three ways — and you can watch all three happen at once. Give Patient 1 one more unit of x, from 0.5 to 1.5, everything else fixed. The logit goes up by exactly w = 1.4 — a clean, straight step, from 0.5 to 1.9. The odds get multiplied by $e^{w}$ = e¹·⁴ ≈ 4.055: recompute directly at x=1.5, σ(1.9) = 1/(1+e⁻¹·⁹) ≈ 0.870, so the new odds are 0.870/0.130 ≈ 6.69 — and 6.69/1.65 ≈ 4.05, matching e^1.4 almost exactly (the small gap is just rounding). And the probability itself moved from 0.622 to 0.870 — a jump of 0.248, far more than the same one-unit step would move a probability already near 0.99 or 0.01. One weight, three honest stories: +1.4 to the logit, ×4.05 to the odds, and a curved, context-dependent move in probability.

The second half: what loss do we train it with?

Reach for the obvious loss — mean squared error, the one linear regression uses — and watch it fail. Bring in two more patients, scored with the same w=1.4, b=−0.2, both of whom truly had a heart attack (y=1). Patient 3 has x=2.0: z=1.4×2−0.2=2.6, σ(2.6)≈0.931. The model was basically right, and the squared error shows it: (0.931−1)² ≈ 0.005 — tiny. Patient 4 has x=−2.0: z=1.4×(−2)−0.2=−3.0, σ(−3.0)≈0.047. The model insisted this person was low-risk, about someone who was not — confidently, badly wrong. Yet the squared error is only (0.047−1)² ≈ 0.908. Push the prediction even further wrong, down to 0.0001, and MSE barely moves at all — (0.0001−1)² ≈ 0.9998, essentially the same number. MSE has a hard ceiling at 1 no matter how confidently wrong the model gets.

That is the whole problem. A loss is the *cost we attach to being wrong* — it is how we tell the model how badly it messed up. MSE tells the model that a confident disaster (0.047, or 0.0001, when the truth is 1) costs about the same as any other bad miss. So the model has no reason to fix its worst mistakes: the loss never screams past a certain point. The signal is too flat to be any use.

Log loss (also called cross-entropy) fixes this by making the cost blow up as a confident prediction turns out wrong, with no ceiling at all:

$L = -[\,y\log(\hat{y}) + (1-y)\log(1-\hat{y})\,]$

Because y is 0 or 1, only one of the two terms is ever active. For Patient 3 (y=1) the loss is just $-\log(\hat{y})$ = −log(0.931) ≈ 0.07 — a gentle cost for a basically-correct call. For Patient 4, −log(0.047) ≈ 3.05 — over 40× larger, for a prediction that was only about 14× further from the truth in raw probability terms (Patient 3 missed the truth by 1−0.931=0.069, Patient 4 by 1−0.047=0.953; 0.953/0.069 ≈ 13.8). And unlike MSE, log loss keeps climbing as the prediction gets worse: at 0.0001 it would be −log(0.0001) ≈ 9.21, still three times Patient 4's cost, with no ceiling in sight. Log loss punishes confident wrongness without any ceiling, which is exactly the message the model needs to hear. That is why we train classification with log loss, not MSE.

Under the hood (the deeper why)

There is a cleaner reason log loss wins, and you can see it in the gradient. Work out how log loss changes as you nudge the logit z, and the messy sigmoid-slope term cancels out perfectly, leaving just $\partial L/\partial z = \hat{y} - y$ — the plain prediction error. Check it on Patient 4: ŷ−y = 0.047−1 = −0.953 — nearly the maximum possible gradient magnitude, exactly when the model most needs correcting. MSE-with-a-sigmoid instead leaves behind an extra $σ(z)(1-σ(z))$ factor: for Patient 4 that's 0.047×0.953 ≈ 0.045, so the true MSE gradient $2(hat{y}-y)σ(z)(1-σ(z))$ works out to ≈0.086 — the gradient shrinks to about 9% of log loss's, precisely when the model is most confident and most wrong, so it barely learns from its worst mistakes. Log loss keeps a full-strength gradient no matter how wrong the model is.

Two failure modes are worth knowing. First, perfect separation: if some feature splits the two classes cleanly in the training data, the model can keep making its weights bigger to push every prediction toward a hard 0 or 1, and the weights run off toward infinity — training never settles (watch for exploding weights or a loss that turns into NaN). A small L2 penalty caps the weights and brings back a finite answer. Second, logistic regression tends to come out well-calibrated when the model is correctly specified: because it is trained to give high probability to what actually happened, a predicted 0.7 often really does mean about 70% in reality — something trees, SVMs, and boosting do not give you for free. But "well-calibrated" is a tendency, not a guarantee: heavy regularisation, class imbalance, a mis-specified model, or a shift between training and serving data can all break it, so you still verify calibration rather than assume it.

And it stretches past two classes: swap the sigmoid for the softmax, which turns a whole set of logits into probabilities that add up to 1, and train it with the same log-loss idea. The boundary it draws stays straight — a line in 2D, a flat plane in higher dimensions — so to bend it you must add curved or interaction features yourself. One practical habit: because the L2 penalty judges weights by size, standardise your features first, or a feature measured in the millions gets penalised on a completely different scale from one measured in single digits.

Reading the weights the way a statistician does: odds ratios

We said a one-unit bump in a feature multiplies the odds by $e^{w}$. That number, $e^{w}$, is the odds ratio, and it is how logistic regression coefficients get reported in medicine and credit — "smokers have 2.3× the odds." Our own model reports it too: e^1.4 ≈ 4.05, so "one standard deviation of cholesterol multiplies the odds of a heart attack by about 4×" is the plain-English readout of w=1.4. Just like linear regression, each coefficient carries a standard error, so you can put a confidence interval around the odds ratio and a p-value on whether it differs from 1 (an odds ratio of 1 means "no effect"). This is the inference layer for classification. And the same tooling split applies: scikit-learn hands you the coefficients but not p-values or intervals — for those you use statsmodels' Logit on an unpenalised fit.

The threshold is a business decision, not 0.5

Logistic regression's real output is a *probability*. Turning that probability into an action — flag this transaction, approve this loan — needs a threshold, and 0.5 is almost never the right one. Recall Patient 4: predicted probability 0.047, and yet y=1 — a real heart attack. At the default 0.5 threshold, this patient is called "low risk" and sent home: a false negative, and in this domain a costly one. The right threshold comes from the *cost of each mistake*. In fraud you can only review, say, 500 alerts a day, so you set the threshold to fill that queue with the highest-risk cases (a precision@K problem). In cancer or cardiac screening a missed case like Patient 4's is far worse than a false alarm, so you deliberately drop the threshold — flagging anyone above, say, 0.03 instead of 0.5 — to buy recall, even though that means more false alarms among the genuinely low-risk patients. In lending the costs are literally dollars. Separate the two steps cleanly: the model estimates probability, and *you* choose the decision threshold from the business costs.

When one class is rare

If only 1% of transactions are fraud, a model that predicts "not fraud" every time is 99% accurate and completely useless — which is why accuracy is the wrong metric under imbalance. Three fixes work together. Weight the rare class more heavily in the loss (`class_weight='balanced'` in scikit-learn), so each rare example counts for more. Move the threshold, as above. And judge the model with the right curve: PR-AUC (precision-recall) is far more informative than ROC-AUC when positives are scarce, because ROC-AUC can look flattering while the model still floods you with false positives.

The practical knobs: C, penalties, and solvers

Regularisation is not optional trivia here — it is how you control overfitting and tame perfect separation. One confusing detail trips people up: scikit-learn's `C` is the inverse of the penalty strength, so *smaller C means stronger* regularisation (C = 1/λ). You also choose the penalty type — L2 (shrink weights), L1 (drive some to exactly zero for feature selection), or Elastic Net (a blend) — and the penalty must match the solver: L1 and Elastic Net need a solver like `saga`, while the default `lbfgs` only does L2. Interview-ready summary: C = 1/λ, L1/L2/Elastic Net, and saga is the one solver that does them all.

More than two classes

Two ways to go past yes/no. One-vs-rest trains one binary logistic model per class ("this class or not") and picks the highest scorer — simple, and each model is independently interpretable. Multinomial (softmax) logistic regression trains a single model over all classes at once, with probabilities that sum to 1, and is usually better calibrated across classes. scikit-learn supports both; multinomial is the default for most solvers.

Making the straight boundary bend

The decision boundary logistic regression draws is *linear* in whatever feature space you give it — a line, a plane, a hyperplane. That is a real limit, but also a lever: you make the model as expressive as you like by *engineering the features*. Add interaction terms (age × blood_pressure) to let features combine, polynomial or spline terms to let a feature curve, and binning to let it jump in steps. Done well, this keeps the interpretability and calibration of logistic regression while letting it fit relationships a raw straight line never could — often you reach for a heavier model only after these run out.

Key points

• What logistic regression really is: a linear equation that predicts the log-odds, and a sigmoid that turns that into a probability. Use it as your first model for any yes/no question where you want a probability you can trust, not just a label — fraud, churn, default, click-through. Its coefficients read cleanly: with w=1.4, b=−0.2, a patient at x=0.5 gets logit z=0.5 and probability σ(0.5)≈0.622; nudge x up by one unit and the logit rises by exactly 1.4 while the odds multiply by e^1.4≈4.05. It is fast, interpretable, and — uniquely among the common classifiers — calibrated out of the box. Reach for something heavier only when the boundary is clearly non-linear or the features interact in ways a straight line cannot capture.
• The trap that stops the model learning: training with MSE instead of log loss. MSE caps the penalty for a confident wrong answer at around 1 — a prediction of 0.047 against a true label of 1 costs only ≈0.908, and pushing it further to 0.0001 barely moves that to ≈0.9998 — so the model shrugs off its worst mistakes; its gradient ($2(hat y-y)σ(z)(1−σ(z)) ≈ 0.086$ in that case, versus log loss's ≈0.953) is only about 9% as large, precisely when the prediction is most confidently wrong, so it barely updates. Log loss on the same case costs ≈3.05, climbing toward infinity as the prediction gets worse, with a gradient (ŷ−y ≈ −0.953) that stays full-strength. Always train classification with cross-entropy. Watch too for perfect separation: a feature that splits the classes cleanly drives the weights toward infinity — a little L2 (in scikit-learn, a lower C) reins them back in.
• The diagnostic: read the reliability diagram — when the model says 0.7, is the real rate about 70%? Logistic regression starts well-calibrated, but strong regularisation shrinks the logits and pulls probabilities toward the middle, and class imbalance can distort them. On a held-out set, bucket the predictions and compare each bucket's predicted probability against its actual positive rate. If the model says 0.8 where the truth is 0.55, it is overconfident — fix it with Platt scaling or isotonic regression fit on a *separate* calibration set, never the training set. And standardise features before fitting, since L2 is scale-sensitive.
• The output is a probability; the threshold is a separate business decision — and under imbalance, 0.5 and accuracy both betray you. Let the model estimate probability, then choose the action threshold from the cost of each error: fill a fixed review queue (precision@K) for fraud, buy recall for cancer screening, weigh dollars for lending. When the positive class is rare, accuracy is meaningless (predict-the-majority scores 99%), so weight the rare class (`class_weight='balanced'`), move the threshold, and judge with PR-AUC rather than ROC-AUC, which flatters models that still spew false positives.
• Know the practical knobs cold: C is inverse regularisation, and the penalty must match the solver. In scikit-learn smaller `C` means *stronger* regularisation (C = 1/λ). Pick the penalty — L2 (shrink), L1 (sparse/select), or Elastic Net (blend) — and remember L1 and Elastic Net need a solver like `saga`; the default `lbfgs` only does L2. For more than two classes, one-vs-rest trains an independent binary model per class while multinomial (softmax) trains one joint model with probabilities summing to 1 and is usually better calibrated across classes.
• The boundary is linear in your feature space — so engineer the features to make it bend, and read coefficients as odds ratios. Logistic regression can only draw a straight boundary in whatever space you hand it, so add interactions, polynomial, spline, or binned features to fit curved and combined relationships while keeping its interpretability. And report coefficients as odds ratios ($e^{w}$) with confidence intervals and p-values — an odds ratio of 1 means no effect — using statsmodels when you need the inference layer scikit-learn omits.

Recap

• Logistic regression = linear equation predicts the log-odds, sigmoid turns it into a probability in (0,1). Example: w=1.4,b=−0.2, x=0.5 → z=0.5 → σ(0.5)≈0.622.
• One weight, three readings: +w to the log-odds, ×e^w to the odds (e^1.4≈4.05), a curved move in probability (0.622→0.870 for a +1 step).
• Train with log loss, not MSE. On a confidently-wrong case (ŷ=0.047, y=1): MSE≈0.908 (capped near 1), log loss≈3.05 (unbounded) — and the gradient gap (0.086 vs 0.953) is why MSE stalls.
• Output is a probability; threshold is a separate business call. A 0.047-probability true positive is a false negative at threshold 0.5 — under imbalance, 0.5 and accuracy both betray you.
• Knobs: C = inverse regularisation (small C = strong penalty); penalty must match the solver.
• Boundary is linear in feature space — engineer features to bend it; read coefficients as odds ratios.

Check your understanding