Convolutional Neural Networks

Convolution mechanics, pooling, receptive field, translation equivariance

Everything so far — activation choice, normalisation, the optimiser's step size — assumed a network of generic flat layers, and said nothing about whether *flat* is even the right shape for the input. For an image, it isn't. Take a 28×28 MNIST digit — 784 raw pixels. Feed them into a flat linear layer of 128 units and you need 784×128 = 100,352 parameters just for that one layer. Worse than the parameter count: the model treats pixel (3,4) and pixel (3,5) as completely independent inputs. Nothing in the architecture says these two pixels are neighbours, that they participate in the same local edge, or that an edge at position (3,4) is the same *kind* of feature as an edge at position (15,20). If the model wants to recognise "a horizontal edge," it has to separately learn that concept, from scratch, at every one of the 784 positions it might appear.

Picture a stencil — a small cut-out pattern you can lay over any part of a larger surface and trace through. Cut the stencil for a horizontal edge once, and you can hold it up to *any* patch of the image; the same nine numbers on the stencil test for "is there a horizontal edge here" whether "here" is the top-left corner or the dead centre. Now picture the flat linear layer's alternative: instead of one reusable stencil, it hand-draws a *separate* horizontal-edge detector at every single position, never reusing the shape it already learned three pixels over. That is the structural gap a flat layer can't close on its own — it has no notion that "here" and "three pixels to the right" might want the same test.

Convolution is that stencil, made mathematical. A 3×3 filter slides across the image, computing a dot product at every position: the same 9 weights applied at (3,4), at (15,20), and at every other location. That is weight sharing — the filter has 9 parameters regardless of image size. Learn a horizontal-edge detector once and it fires on horizontal edges *everywhere*, because the same weights do the computation everywhere — a single 3×3 filter does with 9 numbers what the flat layer above needed 100K numbers to attempt. Pooling takes a filter's output over a small spatial region and keeps only the maximum: if the edge appeared slightly left or slightly right within that region, the pooled value is the same either way — a small, deliberate loss of exact position in exchange for *position invariance*. Stack several convolutional layers and a hierarchy emerges: the first layer detects edges, the second combines edges into corners and curves, the third into shapes, the fourth into objects. Each deeper neuron's receptive field — the patch of the *original* image its value depends on — grows with every convolution it sits behind: a neuron at layer 5 of a 3×3-stride-1 network has a receptive field of 11×11 pixels, spanning a neighbourhood wide enough to cover multiple objects, not just one edge.

NOT this. "CNNs were designed for images." The principle — local patterns plus translation equivariance — applies anywhere locality matters. 1D CNNs classify audio and DNA sequences, where adjacent time steps or nucleotides are locally related. 3D CNNs process video, where nearby frames in time are locally correlated. Graph CNNs extend the idea to molecular structures and social networks. The architecture is not about pixels; it is about exploiting whatever spatial or sequential structure your data has. If your input has the property that neighboring elements are more related than distant elements, a convolutional inductive bias is appropriate. If your input is a bag of features with no meaningful ordering, it is not.

Key points

• Use CNNs over MLPs whenever the input has local structure — weight sharing cuts parameters 10–100× and builds in the right inductive bias. A flat MLP applied to a 224×224 RGB image (150,528 raw pixel values) flattened into a first layer of 1,000 output units needs 150,528×1,000 ≈ 150M parameters just for that one layer. A convolutional layer with 64 filters of size 3×3 over 3 input channels needs 64×9×3 = 1,728 weights (before biases), regardless of image size. The accuracy gain is not from having more parameters — it is from encoding the assumption that local patterns repeat, which is correct for images, audio, and sequences.
• Trap: deepening a CNN without residual connections kills gradients — VGG-19 was state of the art; ResNet-152 beat it by simply adding skip connections. Depth without residuals is not free. With plain convolutions, a 34-layer network was harder to train than an 18-layer network — more depth actually hurt. The skip connection output = F(x) + x gives the gradient a direct path: ∂L/∂x includes the identity term regardless of what F does. This one change unlocked reliable training at 100+ layers. If your CNN depth is above ~10 layers and you are not using residuals, the network is likely training with near-zero gradient in early layers.
• Diagnostic: if early-layer filters look like random noise after training, the network is not learning — check learning rate, initialization, and whether input is normalized. Healthy early filters in a CNN trained on images look like oriented edge detectors and color blobs — not random static. Visualize the first-layer weights after 1 epoch. If they are still indistinguishable from the initialization, the gradient is not reaching them. Candidate causes: learning rate too small for the layer depth, missing or wrong normalization on inputs, or vanishing gradients from missing residuals.

Recap

• Flat layers are blind to spatial structure: an MLP treats pixel (3,4) and (3,5) as completely independent inputs — nothing says they're neighbours or that an edge at one position is the same feature as an edge elsewhere, so it must relearn every feature separately at every position (and a 28×28 digit alone needs ~100K weights for one 128-unit layer).
• Stencil metaphor → convolution: a reusable stencil tested against any patch of a surface is what a flat layer lacks — it hand-draws a separate detector at every position instead of reusing one. Convolution = weight sharing: the same 3×3 filter (9 weights) slides across the whole image, computing the same dot product at every location — learn a horizontal-edge detector once and it fires on edges *everywhere*. That's 9 params where a flat layer needs ~100K.
• Pooling = position invariance: keep only the max activation over a small region, so a feature that appeared slightly left or right gives the same pooled value. The downside is lost localisation — after a few pooling layers you know a feature exists in a region but not exactly where (why segmentation uses skip connections / U-Nets).
• Stacking builds a hierarchy: layer 1 detects edges, layer 2 combines them into corners and curves, layer 3 into shapes, layer 4 into objects — and each deeper neuron's *receptive field* grows, so it "sees" a wider slice of the original image.
• Not just images: the real principle is local patterns + translation equivariance, which applies to 1D (audio, DNA), 3D (video), and graphs (molecules, social networks) — any data where neighbouring elements are more related than distant ones. A bag of unordered features is where it does *not* apply.
• Trap — depth without residuals kills gradients: a plain 34-layer CNN trained *worse* than an 18-layer one; the skip connection (output = F(x) + x) gives the gradient a direct identity path (∂L/∂x keeps an identity term regardless of F), which unlocked reliable training past 100 layers. Above ~10 layers with no residuals, early layers likely train on near-zero gradient.
• Diagnostic: visualise first-layer filters after one epoch — healthy ones look like oriented edge detectors and colour blobs. Still-random static means the gradient isn't reaching them: check learning rate, input normalisation, and missing residuals.

Check your understanding