DL Model Serving

Batching, model parallelism, TorchScript/ONNX, GPU memory, latency SLAs

Quantization shrank the model and sped up each individual forward pass. Serving is the next layer up: how to run that model efficiently under real traffic, not just one request at a time. You put a GPT-2 model behind an API. The obvious way to serve it: take one request, run it, return the answer, take the next. Each forward pass takes 50ms, so you get 20 requests per second. But watch the GPU while this happens — it is 95% idle. A GPU is a machine built to do thousands of multiplications at once, and you are feeding it one request at a time. It is a delivery truck making one trip per parcel.

So fill the truck. Stack 32 requests together and run them in a single forward pass. Because the GPU was mostly empty, those 32 finish in roughly the same 50ms as one did — 640 requests per second from the exact same hardware, no model change at all. Batching is the first and biggest lever in serving.

Dynamic batching: don't wait forever for a full truck

Waiting for exactly 32 requests is bad if traffic is slow — early requests sit around. The production fix is *dynamic batching:* set a small deadline, say 5ms, and run whatever has arrived by then. Ten requests? Batch the ten. Forty? Take a batch and queue the rest. You capture most of the batching gain while keeping the wait bounded. Every serving framework (vLLM, TGI, ONNX Runtime) does this with one config flag.

Generation has a second problem batching can't fix

When an LLM writes a reply one token at a time, producing token number *t* means paying attention to all *t−1* tokens before it. Do this naively and every new token re-computes the attention for every earlier token — the total work grows like n², so a long reply gets punishingly slow near the end.

The fix is the KV cache. The first time you process a token, you compute its attention "key" and "value" and *save them.* Every later token just reuses the saved keys and values instead of recomputing them — the work drops from n² to n. For a 512-token reply that is roughly a 512× cut in attention compute. The cost is memory: those saved tensors pile up with every token and every concurrent user. For LLaMA-7B a single token's cache is 512 KiB (2 × 32 layers × 4096 dims × 2 bytes, for K and V), so a 512-token chat holds exactly 256 MiB — and an 80 GB A100, with the ~14 GB model already loaded, has about 66 GB left for cache: room for roughly 260 such chats before it has to start queuing. This is why long context is expensive: the cache, not the weights, runs you out of memory.

One more generation bottleneck: every token, even the easy ones, pays for a full pass.

Even with the KV cache, generating text one token at a time means paying for a full forward pass through the *big* model for every single token — including the easy, predictable ones ("of," "the," a comma most sentences obviously need). Most of any sentence is exactly that predictable. What if a much smaller, much cheaper model guessed the next few tokens, and the big model only had to *check* those guesses instead of generating each one from scratch?

That's the move: let a small, fast "draft" model guess the next K tokens, then have the big model verify all K *in a single forward pass* — which costs about the same as generating one token, because checking K candidate tokens in parallel is no more expensive than one pass's worth of compute. This is speculative decoding. When the draft guessed right, you got K tokens for the price of one — typical speedups are 2–3×. Wherever the draft guessed wrong, the big model's own prediction at that position is used instead, so correctness is never traded away, only speed. The whole theme of serving: the bottleneck is almost never raw model size — it is how well you keep the GPU full through batching, caching, and quantization.

Key points

• Implement dynamic batching before any other optimization — it is the single highest-leverage change for throughput, often 10–30× improvement with zero accuracy cost. The math is simple: at batch size 1, GPU utilization on a typical LLM inference workload is 5–15%. At batch size 32, it is 60–80%. Matrix multiply FLOP/byte ratio scales with the batch dimension — larger batches use the GPU's memory bandwidth more efficiently. Every serving framework (vLLM, TGI, ONNX Runtime) implements dynamic batching; enabling it takes one configuration flag.
• Trap: KV cache grows linearly with sequence length — at long context (16K+ tokens), KV cache can exceed model weight memory. Set max_sequence_length based on actual P99 request lengths, not the theoretical maximum. For LLaMA-7B with 16K context: KV cache per request = 524KB/token × 16,384 tokens = 8.3GB. On an 80GB A100 with ~60GB available after model weights, that supports 7 concurrent requests at 16K context — versus 200+ at 512 tokens. Profile your actual P99 sequence length from traffic logs before configuring context limits. Allowing 16K context for a workload whose P99 is 1K wastes 16× the KV memory.
• Diagnostic: profile GPU utilization during serving. If under 60%, you are under-batching. If over 95% with high latency, you have over-batched or the model is too large for your SLA — consider quantization or a smaller model. NVIDIA's `nvidia-smi dmon` gives per-second GPU utilization. Under-batching (low utilization) and over-batching (high utilization, high latency) have opposite fixes. The target operating point is 70–85% utilization at your P99 latency budget. Below that: increase max batch size or reduce batch timeout. Above that: add more GPUs, quantize to reduce per-request compute, or use a smaller model.

Recap

• One request at a time wastes the GPU: 50ms/pass = 20 req/s, GPU 95% idle. A GPU wants thousands of multiplies at once.
• Batching = fill the truck: 32 requests in one pass finish in ~the same 50ms → 640 req/s, no model change. The biggest serving lever.
• Dynamic batching: set a small deadline (~5ms), run whatever arrived — most of the gain, bounded wait. One config flag in vLLM/TGI/ONNX.
• KV cache fixes generation's n² problem: save each token's key/value, reuse them → work drops n² → n (~512× for a 512-token reply).
• KV cache cost is memory: ~524KB/token for LLaMA-7B → ~256MB per 512-token chat; an 80GB A100 fits only ~300 chats. Long context runs you out of memory — the cache, not the weights.
• Speculative decoding: small draft model guesses K tokens, big model verifies all K in one pass → 2–3× when the guess is right.
• Diagnostic: target 70–85% GPU utilization at P99 budget — under 60% = under-batching; over 95% with high latency = over-batched or model too big.

Check your understanding