Quantization from First Principles: What FP16 Actually Throws Away

Every DL serving tutorial tells you to use FP16 to halve your memory and double your throughput. Few explain what you're actually sacrificing — and when that sacrifice breaks your model. Quantization isn't free. Here's the bit-level mental model that lets you reason about when it's safe and when it's not.

Quantization is the practice of representing neural network weights and activations with fewer bits. FP32 → FP16 → INT8 → INT4. Each step halves memory. Each step introduces approximation error. Whether that error matters depends on the specific model and the specific operation.

The bit layout of floating point

FP32 uses 32 bits: 1 sign bit, 8 exponent bits, 23 mantissa bits. FP16 uses 16 bits: 1 sign bit, 5 exponent bits, 10 mantissa bits. BF16 uses 16 bits: 1 sign bit, 8 exponent bits, 7 mantissa bits.

The exponent bits determine range: the maximum representable value. The mantissa bits determine precision: how finely values within that range are represented.

What FP16 throws away: precision

By reducing the mantissa from 23 bits to 10, FP16 can represent the same range of values as FP32, but with lower precision within that range. The relative error of any FP16 value is approximately 2× higher than FP32.

For most neural network operations, this is fine. Weights after training are typically distributed in [-1, 1] with gradients of similar magnitude. The precision loss of FP16 doesn't materially affect inference quality.

The FP16 failure mode: activation outliers

Some model architectures have activation outliers — individual neurons with values orders of magnitude larger than the typical activation. When you quantize to FP16, the representable range shrinks. If activation values exceed ~65,504 (the FP16 max), they overflow to inf or NaN. Downstream computations collapse.

Large language models are particularly susceptible. LLaMA-2, GPT-NeoX, and similar models have specific attention head dimensions with outlier activations of 100–1000×. This is why naive FP16 inference on large LLMs produces incoherent outputs.

The fix: LLM.int8() (Dettmers et al., 2022) handles this by decomposing the matrix multiplication: outlier features (typically < 0.1% of dimensions) are computed in FP16, the rest in INT8. Full INT8 quantization without this decomposition destroys LLM output quality.

BF16 vs FP16: why the exponent matters for LLMs

BF16 trades mantissa precision for range. It has the same 8 exponent bits as FP32, meaning the same maximum representable value (~3.4 × 10^38). FP16's 5 exponent bits mean a max of ~65,504.

For LLMs, BF16 is almost always preferable to FP16: activation outliers fit in the range without overflow, and the reduced mantissa precision (7 vs 23 bits, relative to FP32) is acceptable for inference. Modern GPU hardware (A100, H100) supports BF16 at the same throughput as FP16.

If your GPU supports BF16 (Ampere and later for NVIDIA), use BF16, not FP16, for LLM inference.

INT8 and INT4: dynamic range compression

INT8 quantization represents values as 8-bit integers in the range [-128, 127]. There's no concept of exponent — the numeric range is fixed. Quantization maps the float range of each tensor to this fixed range.

The scale factor: `x_int = round(x_float / scale)`, where `scale = max(|x|) / 127`. The scale is chosen per-tensor (per-tensor quantization) or per-channel (per-channel quantization).

Per-tensor quantization uses a single scale for the entire weight matrix. If the matrix has a few very large values, the scale is large, and small values lose all precision. Per-channel quantization uses a separate scale per output channel — more expensive but much better quality.

INT4 (4-bit) halves the memory of INT8. At 4 bits per weight, a 7B parameter LLM fits in ~3.5GB. The quality cost is significant but surprisingly tolerable for inference when combined with mixed-precision (some layers in higher precision) and groupwise quantization (separate scales per group of weights).

The practical decision matrix

| Scenario | Recommendation | |----------|----------------| | Transformer inference, Ampere+ GPU | BF16 | | Transformer inference, older GPU | FP16 with overflow checks | | LLM inference, memory-constrained | INT8 with LLM.int8() | | LLM inference, extreme memory constraint | INT4 with GPTQ or AWQ | | CNN inference, production serving | INT8 per-channel, calibrated on representative data | | Training | BF16 mixed precision (FP32 master weights, BF16 compute) |

What calibration actually does

Static INT8 quantization requires a calibration dataset: a representative sample of inputs run through the model before quantization. Calibration collects the activation distributions at each layer, which are used to choose the optimal scale factors.

A calibration dataset that doesn't represent your production distribution will produce poor scale factors, which will produce quality degradation. 100–1000 representative inputs is typically sufficient. The inputs must cover the range of magnitudes you'll see in production.

TensorRT, ONNX Runtime, and llama.cpp all implement calibration-based quantization. Using these tools correctly requires understanding which layers are quantization-sensitive (attention, layer norm) and may need to be kept in higher precision.