Activation Functions

Sigmoid, tanh, ReLU, Leaky ReLU, GELU, Swish — saturation and dying neurons

Backprop's own closing promised this handoff directly: ReLU stops the vanishing-gradient message from shrinking on the way down, but it trades one failure for another — the dead neuron. That's where this module actually starts: not re-deriving why depth stalls (Backprop already showed that), but working out exactly when a ReLU neuron goes silent forever, and what to do about it.

First, the reminder that motivates the swap in the first place. Build a 10-layer network with sigmoid hidden layers, train it for an hour, and the loss barely budges — the last couple of layers learn, the first eight sit frozen.

As you saw in Backprop, a sigmoid's slope tops out at 0.25, and multiplying that in at every layer shrinks the signal to about 0.25¹⁰ ≈ one in a million by layer 10 — the early layers get no usable gradient and never learn. Swap sigmoid for ReLU in every hidden layer, retrain from scratch, and the whole network comes alive: loss drops, every layer updates, it converges. ReLU's slope is a clean 1 for any active neuron, so it passes the backward signal through untouched — no shrinking factor to compound.

The activation function is the little non-linear squash applied after each layer — what lets a network bend space instead of only drawing straight lines, and, as just shown, what decides whether the learning signal survives the trip backward through the layers.

ReLU's own flaw: the dead neuron

Look again at the same slope that just fixed vanishing gradients: ReLU's slope is exactly 1 for positive inputs, and exactly 0 for negative ones. Vanishing gradients were about the signal getting small everywhere at once; the dead neuron is a different failure — the signal goes to *exactly, permanently* zero for one specific neuron. If a neuron's input lands negative for *every* training example — often because one too-large gradient step shoved its bias down — ReLU outputs 0, its own slope there is 0, and it receives zero gradient *forever*. It cannot recover on its own: a slope of zero means no future update ever nudges it back toward positive territory. This isn't rare in practice — a too-high learning rate can silently kill a large share of a network's ReLU neurons within the first epoch, invisible on the loss curve.

Leaky ReLU fixes this cheaply by giving negatives a tiny slope (0.01) instead of a flat zero, so a trickle of gradient always flows and a neuron can climb back to positive territory. GELU goes further with a smooth curve that never fully flatlines and softly gates each input by how positive it is — which is why BERT, GPT, and essentially every modern Transformer use it.

One rule that is not up for debate: the output activation

All of the above is about the *hidden* layers. The *output* activation is a correctness constraint, not a preference. For a probability (binary classification) you must use sigmoid [squashes any real number into the open interval (0,1)]; for a set of class probabilities, softmax [exponentiates each logit and divides by the sum of all the exponentials, so every output lands in (0,1) and the whole set sums to exactly 1]; for a plain number (regression), no activation at all. Putting a ReLU on the output of a classifier would let it emit nonsensical values — you match the output activation to what the answer is supposed to *be*, and that is a rule, not a tuning knob.

Softmax's own trap: confident on data it has never seen

Cross-entropy only stops penalising a prediction once the correct class's logit is far ahead of every other logit, so training keeps pushing the network to widen that gap for as long as it helps the loss. That habit doesn't switch off on an input the network is actually unsure about — an ambiguous test example still gets pushed through the same wide-gap machinery, so softmax reports 99% confidence on something that is genuinely closer to a coin flip. Two fixes, both applied without changing the architecture: temperature scaling — divide every logit by a constant T>1 before the softmax, which shrinks the gap and cools the reported confidence without changing which class wins — and label smoothing — during training, replace the one-hot target (1 for the correct class, 0 for every other) with a softened target (for example 0.9 on the correct class, the remaining 0.1 split across the rest), so cross-entropy stops rewarding an infinitely wide gap in the first place.

Key points

• Use GELU for hidden layers in Transformers and deep MLPs; use ReLU when compute is tight and the network is shallow (under 6 layers); use sigmoid only as an output activation for binary classification. The output activation is a semantic constraint, not a tuning choice: sigmoid for probability outputs, softmax for multi-class distributions, linear for regression targets. Changing the output activation to match the loss function is correctness, not experimentation.
• The production trap: dying ReLU neurons that silently reduce model capacity. A network trained with too-high a learning rate can end up with a large share of its ReLU neurons permanently dead after the first epoch — those neurons contribute nothing to any forward pass but still consume memory and compute. You will not notice from the loss curve alone. Monitor the fraction of dead neurons by checking how many neurons produce exactly zero output across a validation batch. There's no universal cutoff (it depends on the architecture), but a high or rising fraction signals a capacity problem. Fix: lower the learning rate, use a better initialiser, or switch to Leaky ReLU or GELU.
• The diagnostic: gradient norm ratio between first and last hidden layer. Log the mean absolute gradient at each layer's weights after one backward pass. In a healthy 10-layer network with ReLU, the ratio should be within 10× between first and last layer. With sigmoid, expect 10⁶× or more — every layer of sigmoid compresses gradients by 4×. If you see a large ratio with ReLU, dying neurons are the cause: neurons with zero output contribute zero gradient to the weight update for that layer.

Recap

• Picks up from Backprop's own promise: Backprop closed on "ReLU trades vanishing gradients for the dead neuron" — this module starts there, not by re-deriving vanishing gradients from scratch.
• Recall — sigmoid kills gradients: as Backprop showed, its slope is at most 0.25, multiplied in at every layer → ~0.25¹⁰ ≈ one-in-a-million after 10 layers, early layers stop learning. Activation choice is what decides whether the learning signal survives the trip backward through the layers — the whole history of activations is a sequence of gradient-flow fixes.
• ReLU fixes saturation: "keep positives, zero out negatives" gives a clean slope of 1 for any active neuron, so it passes the backward signal through untouched — the same fix that turned the 10-layer sigmoid network above from stalled to fully trainable.
• ReLU's own flaw — the dead neuron — is a different failure than vanishing gradients: not the signal shrinking everywhere, but going *exactly, permanently* zero for one neuron. If a neuron's input is negative for *every* example (often after one too-large gradient step shoves its bias down), it outputs 0, its slope is 0, and it receives zero gradient *forever* — dead and never recovers. Leaky ReLU gives negatives a tiny slope (0.01) so a trickle of gradient always flows; GELU does the same with a smooth curve.
• GELU: a smooth curve that never fully flatlines and softly gates each input by how positive it is (weighting it by its probability of being positive under N(0,1)) — which is why BERT, GPT, and essentially every modern Transformer use it over ReLU.
• Output activation is a correctness constraint, not a preference: all of the above is about *hidden* layers. For the output you must match the answer's type — sigmoid for a probability (binary), softmax for a class distribution, and no activation at all for a plain regression number. Putting ReLU on a classifier's output is a bug, not a tuning choice.
• Diagnostic — dead-neuron fraction: count how many neurons output exactly zero across a validation batch; a high or rising fraction is a capacity problem (a too-high LR can silently kill a large share of ReLUs in one epoch, invisible on the loss curve). Fix by lowering the learning rate, using a better initialiser, or switching to Leaky ReLU / GELU.

Check your understanding