RNNs & LSTMs

Vanishing gradient in sequences, gate mechanisms, hidden state, when to still use them

Convolution's bias — nearby elements are related, and that relationship repeats — extends to 1D sequences too, but only within a fixed-size kernel's reach. Some sequences need a dependency that reaches arbitrarily far back. Try to classify the sentiment of "The movie was not good." Process it token by token with a vanilla RNN. At each step, the hidden state h_t is updated: h_t = tanh(W_h · h_{t-1} + W_x · x_t). Tokenized by word, "not" is the 4th token and "good" is the 5th, so the hidden state that captures "not" is h_4. By the time the model reaches "good" and computes the loss from h_5, the gradient of that loss with respect to h_4 must travel back through exactly 1 Jacobian matrix — one per timestep of separation, and here the two tokens are only one timestep apart. Each Jacobian for the tanh activation has a spectral radius that, on average, is less than 1, so every additional timestep of separation multiplies in another shrinking factor: if each Jacobian contributes a factor of 0.5, a 4-timestep separation shrinks the gradient by 0.5⁴ = 0.0625, and a 20-timestep separation shrinks it by 0.5²⁰ ≈ 10⁻⁶. In a longer review — say one where "not" sits 20 tokens before the word the sentiment hinges on — the signal from that early token cannot reach the loss gradient strongly enough to update the corresponding weights.

The LSTM was designed specifically to defeat this. Rather than passing the gradient only through the hidden state h_t, it adds a cell state C_t with an additive update path: C_t = f_t ⊙ C_{t-1} + i_t ⊙ g_t. The forget gate f_t ∈ (0, 1) decides how much of the previous cell state to keep. The gradient of C_t with respect to C_{t-1} is f_t — and the LSTM can learn to keep f_t near 1 for timesteps where memory should be preserved. When f_t ≈ 1, the gradient flows backward through the cell state unchanged, giving the early token a direct path to the loss. The input gate i_t decides what new information to write to the cell state -- that new information is the candidate cell value g_t = tanh(W_g · [h_{t-1}, x_t]), a tanh-squashed candidate update (the LSTM's analogue of the GRU's new_h_t below), and i_t decides how much of g_t actually gets written in. The output gate o_t decides what to expose as the hidden state h_t = o_t ⊙ tanh(C_t). The GRU achieves similar behavior with three weight matrices instead of the LSTM's four (~25% fewer parameters): a reset gate r_t decides how much of the previous hidden state feeds into a new candidate state, new_h_t = tanh(W · [r_t ⊙ h_{t-1}, x_t]), and an update gate z_t then controls the mix between the old state and that candidate, h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ new_h_t, so z_t near 0 preserves the old state much like the LSTM's forget gate near 1, while z_t near 1 writes in the candidate — empirically comparable performance to the LSTM on most tasks.

NOT this. "Transformers made RNNs obsolete." For offline NLP with the full sequence available, transformers win on almost every benchmark. But RNNs remain the correct tool for streaming inference: when you are processing an audio stream, a live trading feed, or a robotics sensor reading, you do not have the full sequence at inference time. Transformer attention requires all positions to be present simultaneously — O(n²) memory to compute the attention matrix. An RNN processes each new token in O(1) with fixed memory. For sequences beyond ~16K tokens where attention memory becomes prohibitive, or for tasks with strict sequential causality and real-time constraints, the RNN is not a fallback — it is the right architecture.

Key points

• Use LSTMs for sequential tasks where the full sequence is not available at inference time — streaming audio, live trading, real-time sensor processing. Transformer attention requires all positions simultaneously; an LSTM processes each new token in O(1) with fixed memory. That is the decision boundary: if inference is sequential and unbounded, use an LSTM. If inference can wait for the full sequence and length is under ~16K tokens, use a Transformer.
• Trap: vanishing gradient is not fully fixed by LSTMs for all sequence lengths. Beyond ~200 steps even LSTMs struggle — for long-range dependencies in offline settings, attention is strictly better. The forget gate keeps gradients alive by learning f_t ≈ 1, but it is learned under gradient pressure from the task. For dependencies spanning hundreds of steps, the gradient through the cell state path still attenuates — product of 200 forget gate values, each slightly below 1, compounds. LSTMs win over vanilla RNNs at 20–50 steps. For 500+ steps with the full sequence available, use attention.
• Diagnostic: plot gradient norms per time step during backpropagation through time — if the norm at step 1 is < 1e-4 while step T is 1.0, you have vanishing gradients regardless of LSTM gates. Hook into the backward pass and log ‖∂L/∂h_t‖ for each t. A healthy LSTM should show gradient norms decaying by at most ~100× from the last timestep to the first for sequences under 100 steps. Exponential decay is the vanishing gradient signature. Check that forget gate biases are initialized to 1.0 (not 0.0) — a forget bias of 0 means sigmoid(0) = 0.5, which already shrinks the cell state gradient path by half at every step from initialization.
• Trap: LSTMs/GRUs are trained with teacher forcing — the ground-truth previous token is fed in at every decoding step — but at inference time there is no ground truth, so the model must feed back its own predictions. That training/inference mismatch is exposure bias. During training the decoder conditions each step on the correct prior token regardless of what the model itself would have predicted, so every gradient update is computed on the correct trajectory. At test time the model instead conditions on its own last prediction; one wrong token early in the sequence shifts every hidden state downstream onto a trajectory the model never trained on, and errors compound. Two standard fixes: scheduled sampling, which mixes in the model's own predictions during training with increasing probability as training progresses, and professor forcing, which adds an adversarial loss that pushes the free-running (self-fed) hidden-state trajectory to match the teacher-forced one.

Recap

• Vanilla RNN vanishing gradient: to learn that "not" two tokens back flips the sentiment, the gradient must travel back through one Jacobian per timestep, each with spectral radius under 1 (say ~0.5) → 0.5²⁰ ≈ 10⁻⁶ after 20 steps. The early-token signal can't reach the loss strongly enough to update its weights, so long-range dependencies simply aren't learned.
• LSTM cell state = an additive gradient highway: alongside the hidden state it carries a cell state with an *additive* update, C_t = f_t ⊙ C_{t-1} + i_t ⊙ g_t. The gradient of C_t with respect to C_{t-1} is just the forget gate f_t — no per-step shrinking multiplication, unlike the vanilla RNN's Jacobian chain.
• Forget gate near 1 preserves the signal: because ∂C_t/∂C_{t-1} = f_t, the network can *learn* to keep f_t ≈ 1 exactly where memory should be preserved, giving the early token a nearly unobstructed path back to the loss across hundreds of steps.
• GRU: a reset gate (how much past hidden state feeds the new candidate) and an update gate (mix of old state vs. candidate) replace the LSTM's three gates plus candidate weight matrix — three weight matrices instead of the LSTM's four, ~25% fewer parameters, and empirically comparable performance on most tasks — prefer it on smaller datasets or when compute is tight.
• Not obsolete: for offline NLP with the full sequence available, Transformers win almost every benchmark — but RNNs remain correct for *streaming* inference (audio, live feeds, robotics) where you don't have the whole sequence. Attention needs all positions present at once (O(n²) memory); an RNN processes each new token in O(1) with fixed memory.
• Trap — LSTMs don't fully fix vanishing: beyond ~200 steps the product of many forget gates (each just under 1) still compounds toward zero. LSTMs beat vanilla RNNs at 20–50 steps; for 500+ step dependencies with the full sequence available, attention is strictly better.
• Diagnostic: initialise the forget-gate bias to 1.0, not 0 — sigmoid(0)=0.5 already halves the cell-state gradient path at every step from the start. Then log ‖∂L/∂h_t‖ per timestep; exponential decay toward step 1 is the vanishing signature.

Check your understanding