Decide whether to self-host at all

Start with the uncomfortable question: do you need to run this yourself? For most teams below a steady, high-volume workload, a hosted API wins. You pay per token, you get frontier models, and you carry no GPU on-call. Self-hosting with vLLM makes sense when you have specific, defensible reasons: sustained high traffic where per-token pricing exceeds the amortized cost of a GPU running near capacity; data residency or compliance rules that forbid sending prompts to a third party; a fine-tuned open-weight model you need to serve; or predictable latency requirements you can only meet by owning the hardware.

If you are self-hosting to save money on a few thousand requests a day, stop. The GPU sits idle, the ops burden is real, and you will lose. The math flips only when a GPU stays busy. Everything below assumes you have cleared that bar.

Start the server

vLLM ships an OpenAI-compatible server, which is the single most useful thing about it operationally. Your existing client code that points at the OpenAI SDK keeps working; you change the base URL and model name. Launch it with vllm serve:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --served-model-name llama-3.1-8b

That gives you /v1/chat/completions, /v1/completions, and /v1/models on port 8000. Call it exactly like OpenAI:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [{"role": "user", "content": "Summarize PagedAttention in one sentence."}],
    "max_tokens": 128
  }'

Two flags matter early. --max-model-len bounds the context window the server will accept; setting it lower than the model's maximum frees KV cache memory for more concurrent requests. --gpu-memory-utilization (default 0.9) tells vLLM how much of the card to claim for weights plus KV cache. Push it too high and you risk out-of-memory under bursty load; too low and you waste capacity.

Continuous batching and PagedAttention

These two features are why vLLM exists and why it outperforms naive serving. Understand them before you tune anything.

Traditional static batching groups a fixed set of requests, runs them together, and waits for the slowest one to finish before starting the next batch. Generation lengths vary wildly, so short requests sit idle waiting for long ones, and the GPU stalls. Continuous batching (also called in-flight batching) instead schedules at the token level: when one sequence finishes, its slot is immediately filled by a waiting request. The GPU stays saturated. Under concurrent load this is the difference between a card that is busy and one that spends half its time waiting.

PagedAttention solves the memory side. The KV cache — the stored key/value tensors for every token generated so far — is the dominant memory consumer during inference, and its size is unpredictable because you do not know output length in advance. Classic implementations pre-allocate a contiguous block per request sized for the worst case, which fragments memory and wastes most of it. PagedAttention borrows the idea of virtual memory paging: it stores the KV cache in fixed-size, non-contiguous blocks and maps them with a lookup table. That near-eliminates fragmentation and lets many more sequences share the same GPU. It also makes prefix sharing cheap, so common system prompts across requests reuse the same cache blocks. You do not configure these; they are on by default. But knowing they exist tells you why raising concurrency often costs less memory than you expect, and why long outputs, not request count, are usually what pressures the cache.

Quantization: smaller weights, real tradeoffs

Quantization shrinks model weights from 16-bit down to 8-bit or 4-bit so a model fits on smaller or fewer GPUs and often runs faster due to reduced memory bandwidth. The honest framing: you trade some quality for capacity, and the amount varies by model and task. Measure it on your own evals, not on a leaderboard.

  • AWQ and GPTQ are 4-bit weight-only schemes. They dramatically cut memory and are well supported in vLLM. Quality loss is usually modest but non-zero; verify on tasks that matter to you, especially structured output and tool calling, which are more sensitive to degradation than freeform chat.
  • FP8 (8-bit floating point) is the middle ground on newer hardware (Hopper and later). It keeps more precision than 4-bit and is a strong default when your GPU supports it.

Point vLLM at a pre-quantized checkpoint or set the quantization method explicitly:

vllm serve TheModel/Llama-3.1-8B-Instruct-AWQ --quantization awq

Rule of thumb: reach for quantization when memory is the binding constraint, not reflexively. If your model already fits and the GPU is throughput-bound, quantization buys less than you think and may cost quality you did not need to spend.

Tensor parallelism and multi-GPU

When a model will not fit on one card, or when you want more throughput per replica, shard it across GPUs with tensor parallelism:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.92

--tensor-parallel-size splits each layer's matrices across N GPUs in one node. Set it to fit the model, then measure — parallelism adds cross-GPU communication overhead, so more GPUs is not linearly more throughput. For models spanning multiple nodes, combine with --pipeline-parallel-size, but stay on a single node while you can; NVLink beats network interconnect by a wide margin. Prefer running more single-GPU replicas over one heavily-sharded replica when the model fits on one card, because independent replicas scale cleanly and fail independently.

Autoscaling on Kubernetes

Run vLLM as a Deployment behind a Service. The wrinkle is that CPU-based autoscaling is useless here — the bottleneck is the GPU, not the CPU. Scale on GPU-aware signals instead: request queue depth, time-to-first-token, or GPU utilization exported to Prometheus, wired into a HorizontalPodAutoscaler via custom or external metrics.

Practical guidance that saves outages:

  • Pin one GPU per pod with resource limits (nvidia.com/gpu: 1) and let the replica count scale, rather than packing many models per card.
  • Set generous readiness probes. Model weights take tens of seconds to minutes to load; a pod that is not ready must not receive traffic, and an aggressive liveness probe will kill a pod mid-load and loop forever.
  • Expect slow scale-up. Pulling a multi-gigabyte image and loading weights means new replicas take minutes, so scale on leading indicators (queue depth) and keep headroom. Pre-warmed nodes or a baseline replica count absorb bursts that reactive scaling cannot.
  • Cap max replicas to your GPU quota so the HPA does not wedge on unschedulable pods.

Observability

You cannot operate what you cannot see, and inference has failure modes that generic app metrics miss. vLLM exposes a Prometheus /metrics endpoint — scrape it. The metrics that actually drive decisions:

  • Time to first token (TTFT) and inter-token latency — what users feel. Track percentiles, not averages; the tail is where complaints come from.
  • Throughput in tokens per second, generation and prompt separately.
  • Running vs. waiting requests and queue time — the earliest signal that you are under-provisioned and the best autoscaling trigger.
  • KV cache utilization — when this saturates, requests queue or get preempted; it is your capacity ceiling made visible.
  • GPU utilization and memory from DCGM or the node exporter, correlated with the above.

Wire these into your existing stack the same way you would any service. The general playbook in how to monitor LLM apps in production applies directly, layered on top of these server-level signals.

Where this leaves you

vLLM is the right tool once self-hosting is justified: an OpenAI-compatible endpoint, continuous batching and PagedAttention doing the hard work by default, and clear knobs for quantization, parallelism, and scale. Choosing between engines is its own decision — the vLLM vs. TGI comparison walks the tradeoffs. If you are serving a custom model, pair this with how to fine-tune an LLM, and if your outputs feed downstream automation, put guardrails in front of them before you ship. Deploy, measure against real targets, and tune from evidence rather than defaults.