RNNs and LSTMs: What the Gates Are Actually Solving

The LSTM was not designed to be clever. It was designed to survive backpropagation through hundreds of time steps without losing its gradient signal. The cell state and the three gates are a direct engineering solution to a specific mathematical problem. Understanding that problem makes LSTMs obvious in retrospect — and makes it clear why attention still had to replace them.

Before attention, sequences were modelled with recurrent networks. The intuition was natural: process the sequence one token at a time, maintain a hidden state that carries information forward, update it at each step. The reality was that training these networks on long sequences was nearly impossible. The LSTM solved most of the problem. Then attention made the problem irrelevant.

The RNN: the simple version

h_t = tanh(W_h * h_{t-1} + W_x * x_t + b). At each timestep, the hidden state h_t is a nonlinear function of the previous hidden state and the current input. After N steps, h_N contains (in principle) all information from the sequence. The same weight matrices W_h and W_x are used at every step.

Why vanilla RNNs fail: BPTT and vanishing gradients

Training an RNN requires backpropagation through time (BPTT): unroll the computation graph across all N timesteps and backpropagate. The gradient of the loss with respect to h_1 involves the product of N Jacobians: ∂h_N/∂h_1 = ∂h_N/∂h_{N-1} * ∂h_{N-1}/∂h_{N-2} * ... * ∂h_2/∂h_1.

The Jacobian ∂h_t/∂h_{t-1} = diag(1 - h_t^2) * W_h (for tanh). Its magnitude is determined by the singular values of W_h and the tanh derivative (maximum 1, typically less). For long sequences, this product either vanishes to zero (gradients from early steps are lost — the network cannot learn long-range dependencies) or explodes (numerically unstable training). Gradient clipping handles explosion. Vanishing is harder to fix.

The LSTM: a cell state highway

The LSTM (Hochreiter & Schmidhuber, 1997) adds a cell state c_t alongside the hidden state h_t. The critical design choice: the cell state is updated via addition, not multiplication. c_t = f_t * c_{t-1} + i_t * g_t.

This addition means gradients flow back through the cell state without being multiplied by a Jacobian at each step — just added. It is the same principle as ResNet's skip connection applied across time.

The three gates

The gates are scalar-valued (after sigmoid) and learned. They control information flow.

Forget gate: f_t = σ(W_f * [h_{t-1}, x_t] + b_f). Values near 0 erase the corresponding cell state dimension; values near 1 preserve it. The network learns when to forget: short-term context should clear old information, long-term dependencies should preserve it.

Input gate: i_t = σ(W_i * [h_{t-1}, x_t] + b_i). Controls how much of the candidate cell update g_t = tanh(W_g * [h_{t-1}, x_t] + b_g) actually modifies the cell state. Allows selective writing.

Output gate: o_t = σ(W_o * [h_{t-1}, x_t] + b_o). Controls how much of the cell state c_t (after tanh) is exposed as the hidden state h_t. Allows selective reading.

What LSTMs solve and what they do not

LSTMs can learn dependencies over hundreds of steps where vanilla RNNs fail. For many sequence tasks (language modelling, speech recognition, time series), they were state of the art from 1997 to 2017.

What they do not solve: serial computation. To compute h_t, you need h_{t-1}. The computation cannot be parallelised across time steps. A 1000-step sequence requires 1000 sequential LSTM calls. On GPU, where parallelism across sequences in a batch is exploited, this is workable. But it is a fundamental bottleneck that grows with sequence length.

Attention has O(n^2) memory but O(1) sequential operations: all positions are computed simultaneously. For the sequence lengths common in language (up to a few thousand tokens), Transformers are both faster to train and better at capturing long-range dependencies. LSTMs remain competitive on time-series tasks with short sequences and when explicit temporal order matters.

Try on Colab: train a character-level LSTM language model on tiny Shakespeare. Then replace the LSTM with a single-layer Transformer. Compare training speed per epoch and validation loss at 10 epochs. The Transformer will likely reach lower loss faster despite having a similar parameter count.