Neural Network Fundamentals

Hidden layers, universal approximation, depth vs width, XOR

Take two on/off inputs, x₁ and x₂, and try to learn XOR: output 1 when *exactly one* input is on, and 0 otherwise. Plot the four cases on a grid and try to separate the 1s from the 0s with a single straight line. You cannot — the 1s sit on one diagonal, the 0s on the other. This is not a data problem or a tuning problem; it is a hard limit. A single linear model can only draw *one* straight boundary, and XOR needs the space bent. And here is the thing: *every* problem a neural network solves is, deep down, this same problem — the input cannot be split as-is, so the model has to reshape it first.

One hidden layer is the fix

Slip a layer of two neurons between the inputs and the output, pass each through a non-linear squash, and suddenly the network can draw curved, bent, folded boundaries. For XOR: one hidden neuron learns to fire whenever *either* input is on — an OR gate, active on three of the four cases — and the other fires only when *both* are on — an AND gate, active on just one case. Neither neuron alone can fire only on the "exactly one on" cases: that pattern is XOR itself, and no single linear-plus-squash unit can separate it (push the weights high enough to fire on (0,1) and (1,0) and you necessarily also fire on (1,1)). It's the *combination* — OR minus AND, i.e. OR-and-not-AND — that fires only on "exactly one on." The hidden layer has *transformed* the input into a new space where a straight line finally works — and that is what every hidden layer in every network is doing: reshaping the representation until the decision becomes easy. Mechanically, each layer computes a = sigma(Wx + b), where W is shape output-units by input-units -- so a layer with 512 inputs and 256 outputs has a 256x512 weight matrix (131,072 weights) plus one bias per output unit (256), 131,328 parameters total -- that shape convention and parameter count is what every layer in this course means by "the forward pass."

How wide, how deep?

The universal approximation theorem says something reassuring: a network with a *single* hidden layer, given enough neurons, can approximate *any* continuous function as closely as you like. So why go deep at all? Because "enough neurons" in one layer can be astronomically many — sometimes exponential in the number of inputs — while a *second* layer can represent the same function with far fewer neurons. That is the real case for depth: not "deeper is magically more accurate," but "depth lets you represent complex functions far more *efficiently*, with fewer parameters." For structured inputs like images this efficiency shows up as hierarchy: each layer can compose the previous layer's features into something more complex — edges combine into textures, textures into object parts, parts into whole objects — and a single very wide layer that maps pixels straight to output units has no way to reuse a detected edge across multiple higher-level features the way stacked layers do. Hierarchical composition and parameter-efficiency are the same underlying argument for depth, just visible in different domains.

But — and this matters — the theorem only says a good solution *exists* at depth. It says nothing about whether gradient descent will *find* it. A careless 10-layer network can have its early layers starved of gradient (the vanishing-gradient problem) and simply not learn. That is why so much of deep learning — better activations, normalisation, residual connections — exists purely to make depth *trainable*. Depth buys efficiency; the rest of the toolkit buys the ability to actually use it.

How does the network actually learn these weights?

Everything above is the *forward pass* — matrix multiplies and squashes, turning an input into a prediction from weights that are already fixed. Training is a loop of four steps, repeated thousands of times: forward pass (compute a prediction from the current weights, exactly as above), loss (a single number scoring how wrong that prediction was — mean squared error for a real-valued target, cross-entropy for a class label), backward pass (work out how much each individual weight contributed to that error), and update (nudge every weight a small step in the direction that reduces the loss, then repeat from the top). This module has only covered the first step. The next module, Backpropagation, is entirely about the third — the one that looked hopeless (a network can have a hundred million weights, each seemingly needing its own gradient computation) until a single trick made it cheap. For now, the one thing worth fixing in your head before moving on: the *loss function has to match what the output represents* — get that pairing wrong and the gradients you compute in the next module point nowhere useful.

Key points

• When the problem has structure a single boundary cannot capture. If a scatter plot of your data cannot be separated by a line (or hyperplane in higher dimensions), a single-layer linear model will always fail — not because of tuning, but structurally. Start with one hidden layer of 8–64 neurons. Add depth only when you have verified the model is capacity-limited on training data, not when validation accuracy is poor (that is a regularisation or data problem).
• The production trap: confusing depth with performance. The most common mistake is adding layers when training loss has stalled. Before adding depth, verify that early layers are receiving gradient signal: hook into the backward pass and log gradient norms per layer. A ratio of 1000:1 between last-layer and first-layer gradient norms means your depth is wasted — those early layers are not learning. Adding more layers worsens this. Fix the gradient flow (ReLU, residual connections, better initialisation) before increasing depth.
• The diagnostic: overfit a single training batch. Before any hyperparameter search, take one mini-batch and train on it alone with no regularisation. A correct model should reach near-zero loss within 100–200 steps. If it does not, the model cannot learn anything — the architecture, loss function, or output activation is wrong. This test rules out implementation bugs before you spend hours tuning a broken model.
• Match the loss to the output, and the output activation to the loss. A real-valued target (regression) pairs with mean squared error and no output activation; a class label (classification) pairs with cross-entropy and sigmoid (2 classes) or softmax (multi-class) on the output. This pairing is a correctness rule, not a tuning choice — the forward pass above computes a prediction, but it's the loss that turns "how wrong was that" into the single number the next module's backward pass differentiates.

Recap

• XOR = the whole game: plot the four cases and no single straight line separates the 1s (exactly-one-on) from the 0s (both-on / both-off) — they sit on opposite diagonals. This is a structural limit of a linear model, not a data or tuning problem, and every problem a network solves is deep down this same "the input can't be split as-is" problem.
• One hidden layer is the fix: slip in a layer of non-linear neurons and the network can bend, fold, and curve the boundary — it *transforms* the input into a new space where a straight line finally separates the classes. That reshaping is exactly what every hidden layer in every network is doing.
• Universal approximation theorem: one hidden layer with enough neurons can approximate *any* continuous function arbitrarily well. The catch: "enough" can be astronomically many (exponential in the inputs), whereas a second layer represents the same function with far fewer neurons — so depth is about *efficiency*, not reachability.
• Depth buys efficiency, not free accuracy: the theorem only says a good solution *exists* at depth — it says nothing about whether gradient descent will *find* it. A careless deep net can starve its early layers of gradient and simply not learn, which is why activations, normalisation, and residual connections exist: to make depth *trainable*.
• Training is a 4-step loop, and this module only covers step 1: forward pass (compute a prediction) → loss (one number scoring how wrong it was — MSE for a real target, cross-entropy for a class label) → backward pass (how much each weight contributed to that error) → update (nudge every weight to reduce it), repeat. Backpropagation, next, is entirely about step 3.
• Verify capacity-limited before adding depth: the common mistake is stacking layers when training loss stalls. First hook the backward pass and log per-layer gradient norms — a 1000:1 ratio between last-layer and first-layer norms means the early layers get no signal and the depth is already wasted; more layers make it worse. Fix gradient flow (ReLU, residuals, better init) before going deeper.
• Diagnostic — overfit one batch: before any hyperparameter search, train on a single mini-batch with no regularisation; a correct model should crash to near-zero loss in 100–200 steps. If it can't, the architecture, loss function, or output activation is wrong — this rules out implementation bugs before you waste hours tuning a broken model.

Check your understanding