Serving a large language model in production is a memory-bandwidth problem wearing a compute problem's clothes. Most engineers' first instinct — "the GPU isn't fast enough" — is usually wrong. The GPU spends most of a generation waiting on memory, not doing math. Understanding why is the key that unlocks every serving optimization worth knowing: the KV cache, batching, quantization, and speculative decoding all exist to fight one of two specific bottlenecks.
This explainer is for engineers who run models — via vLLM, TGI, or a managed endpoint — and want to reason about latency, throughput, and cost instead of guessing. If you're choosing a serving engine, vLLM vs TGI compares the two, and how to deploy vLLM in production covers the operational side.
Two phases, two bottlenecks
A single generation has two distinct phases, and they behave completely differently.
Prefill processes your prompt. Every input token is run through the model in one parallel pass, producing the internal state the model needs to start generating. Because all the prompt tokens are processed at once, prefill is highly parallel and compute-bound — it saturates the GPU's matrix-multiply units. Prefill cost scales with prompt length, which is why a long system prompt or a big retrieved context is expensive before a single output token appears. This phase determines your time-to-first-token (TTFT).
Decode generates the output, one token at a time. Each new token depends on all the previous ones, so decode is inherently sequential — you can't generate token 50 before token 49. Here's the counterintuitive part: generating one token requires loading the model's entire weight matrix from GPU memory to do a single small matrix-vector multiply. The arithmetic is trivial; the memory read dominates. Decode is memory-bandwidth-bound. This phase determines your time-per-output-token (TPOT) and thus how fast text streams.
Almost every optimization below targets one phase. If you don't know which phase your bottleneck is in, you're optimizing blind.
The KV cache: the central data structure
Attention is the reason decode could be slow and the KV cache is the reason it isn't. When the model generates each new token, attention needs the keys and values of every previous token. Recomputing them from scratch for every new token would make generation quadratic and hopeless.
The fix is caching: after a token's keys and values are computed once, store them and reuse them. That store is the KV cache, and it turns per-token work from "reprocess the whole sequence" into "process one new token and read the cache." It's the reason decode is O(n) instead of O(n²) per token.
The cache is also the memory monster of LLM serving. Its size grows linearly with sequence length, batch size, number of layers, and model width — a long conversation with a big context can consume gigabytes of GPU memory per request, often dwarfing the model weights themselves. This is the real constraint on how many concurrent requests you can serve: not compute, but how many KV caches fit in memory. Everything about serving scale comes back to KV cache management.
The major advance here was PagedAttention (introduced by vLLM), which borrows virtual-memory paging from operating systems. Instead of reserving one big contiguous block per request — which wastes memory to internal fragmentation and forces you to over-provision for the worst-case length — it stores the cache in fixed-size pages allocated on demand. That slashes waste, lets far more requests share the GPU, and enables cheap prefix sharing (two requests with the same system prompt can share those pages). PagedAttention is a large part of why modern serving throughput jumped.
Continuous batching: keeping the GPU busy
Because decode is memory-bandwidth-bound, running one request at a time wastes the GPU: you pay the full cost of loading the weights to generate a single token for a single user. If you load those same weights and generate a token for many users at once, the expensive memory read is amortized across the whole batch. Batching is close to free throughput — up to the point where you run out of KV-cache memory or compute.
Naive static batching waits to collect a batch, runs it to completion, then starts the next. It's simple and terrible for LLMs, because requests finish at wildly different lengths — one user wants 5 tokens, another wants 800. The whole batch is held hostage by its longest member, and finished slots sit idle.
Continuous batching (a.k.a. in-flight batching) fixes this by operating at the token level. After every decode step, the scheduler evicts any requests that just finished and admits waiting ones into the freed slots. The batch composition changes continuously; the GPU never idles waiting for the slowest request. This is the single biggest throughput win in modern LLM serving and the core feature of vLLM and TGI. Combined with PagedAttention's flexible memory, it's why a well-configured server can serve many times the requests-per-dollar of a naive loop.
Quantization: trading precision for memory and speed
If decode is bottlenecked on loading weights from memory, then shrinking the weights directly speeds it up — fewer bytes to move per token. That's quantization: storing weights (and sometimes activations and the KV cache) in lower precision than the native 16-bit floats.
Common targets are 8-bit and 4-bit integers. 8-bit quantization roughly halves the memory footprint and bandwidth with negligible quality loss in most cases. 4-bit (via schemes like GPTQ, AWQ, or similar) quarters it and lets large models fit on smaller GPUs, at a quality cost that ranges from imperceptible to noticeable depending on the model and method. The mechanics: map the high-precision range onto a small integer range with a learned scale, sometimes calibrated on sample data to protect the weights that matter most.
The honest tradeoffs. Quantization helps most when you're memory-bound — which decode usually is — so it improves both the cost (bigger models on cheaper hardware) and often the latency. But it's not free quality: aggressive 4-bit can degrade reasoning and factual accuracy in ways that only show up on a real evaluation, not on a few eyeballed samples. And speedups are hardware-dependent — you need kernels that actually exploit the low precision, or you save memory without saving time. Always measure quality on your task after quantizing; never assume.
Speculative decoding: parallelizing the sequential phase
Decode is sequential because token n+1 depends on token n — that's the fundamental limit. Speculative decoding is a clever way around it that produces identical output to the base model, no quality loss.
The trick: use a small, cheap "draft" model to guess the next several tokens quickly, then have the big "target" model verify all of them in a single parallel forward pass — the same kind of parallel pass prefill uses. Because the target verifies many guessed tokens at once, and verifying is cheap relative to generating, you accept whatever prefix the target agrees with and only pay full sequential cost when the draft is wrong. When the draft model is good, you generate several tokens per target-model step instead of one, cutting latency substantially.
It works precisely because decode is memory-bound: the target model's single verification pass costs almost the same as generating one token (the expensive part is loading weights, which you do once either way), but it validates several tokens. The costs are real, though: you run and maintain a second model, you spend extra memory on it, and the speedup depends entirely on the draft model's acceptance rate — on hard, unpredictable text the draft is often wrong and the benefit shrinks. Variants like Medusa (extra prediction heads instead of a separate model) trade some of that complexity away.
Throughput versus latency: the trade you can't escape
Every lever above sits on one tension: throughput (tokens or requests per second across all users, which sets your cost-per-token) versus latency (how fast one user gets their answer). You cannot maximize both.
Bigger batches raise throughput — the GPU serves more users per weight-load — but each user waits longer, because their tokens are interleaved with everyone else's and the batch competes for compute. Smaller batches (down to one) minimize a single user's latency but leave the GPU underused and drive cost-per-token up. This is a genuine dial, not a bug to fix.
Which end you tune toward is a product decision. A real-time chat or voice interface lives and dies on TTFT and streaming speed — you'll cap batch size and accept lower utilization to keep responses snappy. A nightly batch job summarizing a million documents cares only about total throughput and cost — you'll push batch size high and ignore per-request latency entirely. Most serving stacks let you set a target and a max batch size to pick your operating point. The mistake is tuning for throughput and then being surprised the chat feels sluggish, or tuning for latency and then being surprised the batch job costs a fortune. Decide which number your application is graded on first, then turn the dials toward it — and validate the result on production-shaped traffic, because synthetic benchmarks rarely predict real latency.