Quantisation & Model Efficiency

INT8 vs FP16, quantisation-aware training vs PTQ, calibration, accuracy tradeoff

Fine-tuning (LoRA or otherwise) answered how to adapt a trained model cheaply. This module answers a different question: how to make that trained model small and fast enough to actually run. A trained model stores every weight as a float32 number — 4 bytes each. GPT-2's 117 million weights take 468 MB just sitting there. On a phone, that is often too big to load and too slow to run. The obvious question: do we really need 4 bytes of precision per weight, or can we get away with less?

We can get away with a lot less. Store each weight as an 8-bit integer instead — 1 byte — and the file shrinks 4× to 117 MB. Better still, most CPUs have special hardware for 8-bit integer math (the same circuitry that makes video and audio codecs fast), so the model also runs 2–4× faster. For a phone, that is often the difference between "can't run this" and "runs smoothly."

How do you turn a float into an 8-bit integer?

An 8-bit integer can only be one of 256 values. So you take the actual range of the weights — say [−0.5, 0.5] — and chop it into 256 evenly spaced buckets. Each weight is rounded to its nearest bucket: `x_int = round((x_float − min) / scale)`, where `scale = (max − min) / 255` — this maps the range's low end to integer 0 and its high end to 255, the full span of an unsigned 8-bit integer. Two nearby floats that land in the same bucket become the same integer. That rounding is the price you pay — a small error per weight.

The catch is *outliers.* If 99.9% of weights sit in [−0.5, 0.5] but one weight is 5.0, the range must stretch to cover it — and now your 256 buckets are spread across a huge span, wasting almost all of them and leaving the common weights with almost no precision. Handling outliers well is the whole game in quantization.

The key move: calibrate on real data

Weights are fixed after training, so their range is known exactly. But *activations* — the numbers flowing between layers — change with every input, and you can't know their range in advance. So you *calibrate*: run 100–1000 real, representative inputs through the model, watch the actual activation ranges at each layer, and set the scale factors from what you see. This is post-training quantization (PTQ) — no retraining, done in minutes. With good calibration, INT8 typically loses under 1% accuracy. Skip calibration — guess the ranges instead of measuring them — and accuracy collapses silently, with no error in the logs. That silent failure is the single most common quantization mistake.

When PTQ isn't enough

Push down to 4-bit and PTQ starts dropping 2–5% accuracy — too many weights crammed into too few buckets. Two fixes. Smarter PTQ (GPTQ, AWQ) protects the weights that matter most and nudges the rest to compensate for rounding, reaching 4-bit at under 1% loss. Or quantization-aware training (QAT): simulate the rounding *during* training so the model learns to place its weights where they round cleanly. QAT recovers the most accuracy but costs a full retraining run — so you reach for it only when 4-bit-and-below quality is critical.

Key points

• Apply INT8 PTQ to every model before production deployment — it is 2–4× faster on CPU, 4× smaller in memory, and nearly free accuracy-wise with proper calibration. ONNX Runtime or TensorRT handles this in a few lines. The two-line version in ONNX Runtime: `quantize_dynamic(model_path, output_path, weight_type=QuantType.QInt8)`. The full version with static calibration: provide a calibration dataset and use `quantize_static`. The static version consistently outperforms dynamic quantization by 0.3–0.8% accuracy because it calibrates activation ranges, not just weight ranges.
• Trap: quantizing without a calibration set causes severe accuracy loss — the scale factors are wrong. Always run calibration on 100–1000 representative examples from your deployment distribution, not random noise. The scale factor must cover the actual activation value range at inference. If calibration is done on random inputs (or skipped), the scale factors are computed from a distribution that does not match deployment. Activations get clipped to the scale range, producing large quantization errors that look like random noise in predictions. The model will have low average accuracy with no error or warning in the serving logs.
• Diagnostic: per-layer quantization error — if one layer shows much higher reconstruction error than others, that layer has outlier activations. Apply mixed-precision (keep that layer in fp16) before resorting to QAT. Quantization frameworks expose per-layer error metrics. A single layer whose int8 reconstruction differs from float32 by more than 2% of the output range is an outlier layer. Mixed-precision keeps that one layer in fp16 while quantizing everything else to int8, recovering most of the accuracy at a small memory cost. This is almost always cheaper than running a full QAT retraining run.

Recap

• FP32 = 4 bytes/weight: GPT-2's 117M weights = 468MB, often too big/slow for a phone.
• INT8 = 1 byte: 4× smaller *and* 2–4× faster (CPUs have integer-math hardware).
• Float → int: `x_int = round((x_float − min) / scale)`, `scale = (max − min) / 255` — 256 buckets spanning [0, 255], rounding is the cost.
• Outliers are the whole game: one weight at 5.0 stretches the range, wasting buckets and starving the common weights of precision.
• Calibrate on real data: activation ranges are input-dependent, so run 100–1000 representative inputs to set scale factors. This is PTQ — no retraining, minutes, typically <1% loss.
• Skip calibration → silent accuracy collapse: no error in the logs. The single most common quantization mistake.
• Below INT8: GPTQ/AWQ reach 4-bit at <1% loss; QAT simulates rounding during training for the best accuracy but costs a full retrain.

Check your understanding