The Transformer Architecture: Why It Beat Everything
"Attention is All You Need" replaced recurrent networks with a parallelizable architecture that scales. But the paper's real contribution is not attention — it is the combination of multi-head attention, residual connections, layer normalisation, and feedforward networks into a block that stacks reliably to any depth. This is what each component contributes and why removing any one breaks the whole.
The Transformer paper (Vaswani et al., 2017) introduced an architecture with no recurrence, no convolution, and no sequential computation dependencies. It processes entire sequences in parallel and scales with compute in a way RNNs could not. Understanding why requires understanding what each component contributes.
The encoder block: four components, each load-bearing
A single Transformer encoder block has four components in sequence: multi-head self-attention, a residual connection with layer normalisation, a position-wise feedforward network, and another residual connection with layer normalisation.
Multi-head self-attention (see Post 54) allows every position to aggregate information from all other positions. It handles the relationship modeling. The feedforward network (two linear layers with a ReLU or GELU between them) applies the same transformation to each position independently. It handles the representation transformation — taking the attended-to information and projecting it into a richer feature space. Removing either one degrades the model. The attention layers alone are good at routing information; the FFN layers are good at transforming it. Both are necessary.
Residual connections: why depth is possible
Every sub-layer output is added to its input: output = LayerNorm(x + Sublayer(x)). This is the same residual connection from ResNet (see Post 52), applied to sequences. The gradient flows directly back through the addition, bypassing the sub-layer. A 12-layer Transformer is stable to train precisely because each layer can contribute incrementally rather than needing to carry the full representational burden.
Layer Normalisation: why not Batch Norm?
Batch Normalisation normalises over the batch dimension. For language models, sequences have variable length, batch sizes are small at inference time (often 1), and the token-level statistics are not as stable as spatial statistics in images. Layer Normalisation normalises over the feature dimension instead — it is computed independently for each token, independently for each example. It works for any batch size and any sequence length, which is why it became the standard for sequence models.
Positional encoding: injecting order without recurrence
Self-attention is permutation-equivariant: shuffling the input tokens shuffles the output in the same way. The model has no inherent sense of position. Positional encodings fix this by adding a position-dependent signal to each token embedding before attention. The original Transformer used sine and cosine functions at different frequencies: PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d)). These functions produce unique encodings for every position, vary smoothly, and allow the model to attend to relative positions via linear combinations. Later models replaced fixed sinusoidal encodings with learned positional embeddings (BERT, GPT) or relative position encodings (RoPE, ALiBi).
The decoder: masked attention and cross-attention
In sequence-to-sequence tasks (translation, summarisation), the decoder generates tokens one at a time but is trained with teacher forcing — the correct output sequence is fed in, and the decoder learns to predict the next token. Masked self-attention prevents position i from attending to positions j > i, enforcing causality during training. Cross-attention lets each decoder position attend to all encoder positions, enabling the decoder to extract relevant source information for each target token it generates.
Why it beat RNNs
RNNs process sequences step by step. The hidden state at step t depends on step t-1, which depends on t-2, and so on. This serialises computation — you cannot parallelise across time steps. Training a 1000-step sequence requires 1000 sequential matrix multiplications before gradients flow back to step 1. Long-range dependencies degrade because gradients must survive this long chain (LSTMs help but do not eliminate the problem).
Transformers process all positions simultaneously. The maximum path length between any two positions is 1 (direct attention). Long-range dependencies are as easy to learn as short-range ones. The full sequence computation is a matrix multiply — parallelisable on GPU. Training a 1000-token sequence takes the same number of sequential steps as training a 10-token sequence.
The trade-off: O(n^2) memory for the attention matrix vs O(n) for RNN hidden states. For the sequence lengths common in NLP (up to a few thousand tokens), the parallelism benefit far outweighs the quadratic memory cost.
Try on Colab: implement a minimal Transformer encoder from scratch — multi-head attention, feedforward, residual + layer norm — and train it on a character-level language modelling task (tiny Shakespeare). Compare training curves and final loss against a vanilla RNN on the same task.