CNNs: What the Layers Are Actually Computing

A convolutional layer is not a black box. It is a sliding dot product applied across space — and once you see it that way, weight sharing, feature maps, receptive fields, and the necessity of skip connections all follow directly from the math. This is the geometry of what a CNN learns.

A convolutional layer applies the same small learnable filter to every position in an input. That one sentence contains most of what makes CNNs work. Everything else — feature maps, translation invariance, hierarchical features, receptive fields — follows from what that operation implies.

A convolution is a sliding dot product

Take a 3×3 filter — a matrix of 9 learnable weights. Slide it across your 2D input. At each position, compute the dot product between the filter weights and the 9 input values underneath. Record the result. The collection of all these scalar results is one feature map.

A dot product is high when the filter and the input patch are aligned — when they look similar. So a filter that has learned to look like a vertical edge will produce high activations wherever vertical edges appear in the input, and low activations everywhere else. The filter is a template; the feature map is a map of where that template matches. This is the substance of what "the network learns filters" means. The learning process — via backpropagation — finds filter weights that produce feature maps useful for the task.

Weight sharing and why it matters

The same 9 weights are used at every spatial position. A network with a 256×256 input and one 3×3 filter has exactly 9 weights for that layer — not 256×256×9. This is weight sharing, and it is why CNNs are tractable for images. It also encodes a strong prior: the patterns that matter in images are translation-invariant. An edge is an edge whether it appears in the upper-left or lower-right of an image. A fully-connected layer would need to relearn the same pattern at every spatial location independently.

Multiple filters produce a volume

In practice, a convolutional layer has N filters, each independently learned. N filters on a single-channel input produce N feature maps. Stack them and you get a 3D output volume: height × width × N channels. The next convolutional layer applies its filters to this entire volume — each filter now spans all N channels. This is how the network moves from detecting simple patterns (edges in single-channel patches) to combining them (detecting a corner = a horizontal edge and a vertical edge appearing together).

Receptive fields: how deep layers see more

The receptive field of a unit is the region of the original input that can influence its value. For the first layer, a unit sees a 3×3 patch. For the second layer, a unit sees a 5×5 patch of the original input — it integrates over a 3×3 neighbourhood of first-layer units, each of which saw 3×3. Add more layers and the receptive field grows. Pooling layers (max pool, average pool) reduce spatial dimensions, which accelerates receptive field growth. By the deep layers of a CNN, individual units have receptive fields spanning most of the input — they are sensitive to global patterns rather than local edges.

The feature hierarchy

Visualising what filters learn in trained CNNs is instructive. Layer 1 filters respond to oriented edges and colour blobs. Layer 2 combines these into textures and corners. Layer 3 detects parts: eyes, wheels, handles. Deep layers respond to semantic concepts regardless of where they appear in the image.

This hierarchy emerges from training, not design. Backpropagation finds, from scratch, that decomposing images into edges → textures → parts → objects is an efficient way to solve visual tasks. The architecture provides the inductive bias that makes this decomposition representable. The data and the loss provide the supervision.

ResNet skip connections: the gradient argument

Deep networks trained without skip connections suffer from gradient degradation even with ReLU. Multiplying gradients through 50 layers, even if each is close to 1, accumulates enough loss that early layers barely update.

Residual connections change the computation from y = F(x) to y = F(x) + x. The gradient of the loss with respect to x is now: d(loss)/d(x) = d(loss)/d(y) * (d(F(x))/d(x) + 1). The +1 term means the gradient flows directly from the output back to x, bypassing however many layers are inside F. Early layers receive a clean gradient regardless of what F learns, enabling stable training at 100+ layers.

The other benefit: a residual block can learn to be the identity (F(x) = 0) if that is optimal. A plain layer cannot turn itself off — it must always transform its input. Residual blocks can selectively apply transformation where useful and pass inputs through unchanged where not.

Try on Colab: load a pretrained ResNet-18 and use GradCAM to visualise which regions of an input image activate the output for a given class. Then manually extract and display the convolutional filters from layer 1 — compare them to the oriented-edge detectors the theory predicts. The match is surprisingly clean even for small models.