DL Training Failure Modes

Loss spikes, NaN gradients, mode collapse, underfitting vs overfitting — debugging

Every module in this course has quietly asked the same question in its own diagnostics — Backprop's gradient norms, Activations' dead-neuron checks, Batch Norm's train/eval mismatch: is the model actually learning, or does it just look like it is? This module closes the sequence by pulling that recurring thread into one systematic discipline — the checks worth running before trusting any model enough to fine-tune, quantize, or serve it.

You start training a new model and the loss just sits there — stuck at log(number of classes), epoch after epoch. The natural instinct is to start turning knobs: a different learning rate, a bigger model, more data. Resist it.

Think of it like an ER doctor's first sixty seconds with a patient: before asking about the specific complaint, check pulse and breathing — the two or three cheap vital signs that rule out the fastest, most catastrophic explanations. A model has its own vital signs, and checking them costs about as little. Before asking "why won't it learn what I want," ask the more basic question: *can this model learn anything at all?* Almost every training bug is easier to catch by answering that first.

Step 1: overfit a single batch

Take one batch. Turn off all regularization. Train on just that one batch for a thousand steps. A working model *must* be able to memorize a handful of examples — the loss should crash to near-zero. If it can't even do that, no hyperparameter will save you; the model is wired wrong. Usual suspects: the loss doesn't match the output (softmax paired with MSE, or cross-entropy fed raw logits), the output layer has the wrong number of classes, the labels are the wrong shape, or a forward-pass bug is zeroing activations. This test takes 60 seconds and rules out every one of those at once. It is the single most valuable habit in debugging deep nets.

Step 2: if that passes but full training won't converge, look at gradient flow

The signal is learning, but maybe it isn't reaching every layer. Log the average gradient size for each layer after a step. In a healthy network the biggest and smallest layer gradients stay within ~10× of each other. See 10,000× instead and the early layers are getting almost nothing — vanishing gradients — and they will not learn no matter how long you train; the fix is ReLU, residual connections, or better initialization, not more epochs. The reverse — early layers with huge gradients — is exploding gradients: clip them (max_norm=1.0) and check your initialization.

Watch both vital-sign checks catch real bugs on one model

Say you're training a small binary classifier — malicious versus benign network traffic — with a 2-unit softmax output and cross-entropy loss. Step 1 shows loss at log(2) ≈ 0.693; step 50 still shows 0.693, unmoved. Following the diagnostic above: the single-batch overfit test also sits frozen at 0.693. You inspect the logits directly across steps on the same fixed input and they never change at all — the output layer is receiving zero gradient. The cause, once you look: the code calls .detach() on the logits before computing the loss, silently cutting the backward pass off at the very last layer. Remove the .detach() call, rerun the single-batch test — loss now crashes to 0.02 in under 200 steps, exactly as the diagnostic predicts for a correctly wired model.

Rerun full training with the fix in place. Loss falls smoothly from 0.693 toward 0.41 over the first few epochs — then, at step 47, it reports NaN. Looking at the gradient norm logged just before the spike: 8.2 two steps earlier, 41.6 one step earlier, then NaN. That is exploding gradients, not corrupted data — the norm was climbing steadily before it overflowed. Adding gradient clipping (clip_grad_norm_, max_norm=1.0) and rerunning: loss now decreases cleanly to 0.19 by epoch 10, no NaN. Two classic failures, one model, one continuous debugging session — not two abstract rules to memorise separately.

A shorter pair worth knowing by the same two-question habit: training loss falling steadily while validation loss stays flat usually means train and validation don't come from the same distribution, or a decision threshold hasn't been tuned on validation data at all. And a NaN that shows up specifically inside a custom loss, rather than from an exploding gradient norm, is often a bare log(0) — add a small ε inside the log before trusting the gradient math further.

And the trap that fools everyone: a falling loss does not mean it's working. Loss can drop steadily while the model learns nothing useful. On an imbalanced dataset, a model that always predicts the majority class hits 95% accuracy and a nicely decreasing loss curve — and is completely useless. So look past the curve: eyeball a few actual predictions, check the confusion matrix (is it just always guessing one class?), and check that gradient norms sit in a sane range (~0.001–10).

Key points

• Always start debugging by overfitting a single batch — if loss does not reach near-zero on 1 example, the model is fundamentally misconfigured before any data issue can matter. Reduce to 1 batch, remove all regularization (dropout, weight decay), train for 1,000 steps. Loss should converge to near-zero. If it does not, the model has a bug — wrong output activation, wrong loss function, dimension mismatch, or a broken forward pass. This test costs 60 seconds and rules out all configuration bugs before you touch hyperparameters.
• Trap: tuning hyperparameters before running sanity checks. A model with a bug can have decreasing loss but be completely wrong — never tune until the single-batch overfit test passes. A cross-entropy loss with softmax output decreasing from 2.3 to 1.8 over 10 epochs looks like progress. It is not, if the model has an off-by-one label error and is learning to predict the class one index above the correct class everywhere. The single-batch overfit test catches this: the model will overfit the single example to near-zero loss, but inspection of the prediction will show the wrong class label being predicted with high confidence.
• Diagnostic: gradient norm logging catches 80% of training bugs — if any layer's gradient norm is 0 or greater than 100, you have found the problem layer. Add a single hook at the start of training, not after the model is already broken. In PyTorch: `param.register_hook(lambda g: print(g.abs().mean()))` on each layer, or use a unified hook that logs per-layer norms to a dictionary. Run it for the first 10 steps. Norm of 0 on a layer: dead neurons (ReLU killing all activations), wrong weight initialization, or a missing gradient path. Norm above 100: exploding gradients, clip with `torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)`. Both are fixable in minutes once localized to the specific layer.

Recap

• Closes the sequence: pulls together the diagnostic thread already running through Backprop's gradient norms, Activations' dead-neuron checks, and Batch Norm's train/eval mismatch into one systematic discipline. ER-doctor metaphor: check pulse and breathing (cheap vital signs) before diagnosing the specific complaint.
• Before "why won't it learn what I want," ask "can it learn anything at all?" Most bugs surface there first — resist turning knobs.
• Step 1 — overfit one batch: turn off regularisation, train on one batch for 1000 steps; a working model must crash to near-zero loss.
• If it can't: loss/output mismatch (softmax+MSE, cross-entropy on raw logits), wrong class count, wrong label shape, or a forward-pass bug. 60-second test rules them all out.
• Step 2 — gradient flow: log per-layer average gradient; healthy layers stay within ~10× of each other. 10,000× = vanishing (fix: ReLU, residuals, init); huge early gradients = exploding (clip max_norm=1.0).
• One worked trace, two bugs, one model: loss frozen at log(2)=0.693 → logits never change → a stray `.detach()` was cutting the backward pass; fixed, single-batch loss crashes to 0.02. Rerun full training → NaN at step 47, gradient norm 8.2→41.6→NaN just before → exploding gradients, not data; `clip_grad_norm_(max_norm=1.0)` fixes it, loss reaches 0.19 by epoch 10.
• Shorter pair: train falling but val flat → distribution mismatch or an untuned threshold; NaN from a custom loss (not an exploding norm) → bare `log(0)`, add ε.
• The trap: a falling loss doesn't mean it works — always-predict-majority hits 95% accuracy on imbalanced data. Check predictions, confusion matrix, and gradient norms (~0.001–10).

Check your understanding