Running a model for one person on a laptop and serving it to a thousand concurrent users are different engineering problems. The first is about fitting weights in memory. The second is about keeping an expensive GPU busy while requests of wildly different lengths arrive at random and each one holds a growing slab of memory. The techniques that solve this, continuous batching and PagedAttention, are the core of modern serving engines like vLLM. This article explains the problem they solve, how they work, and how to run and tune a server in practice.

Why Serving Is Hard

Generation has two phases with different characters:

  • Prefill processes the whole prompt in parallel. It is compute-bound: lots of matrix math, GPU cores busy.
  • Decode generates one token at a time, each step reading all the model weights and the request’s KV cache. It is memory-bandwidth-bound: the GPU spends more time moving data than computing.

Because decode is bandwidth-bound, a GPU handling one request at a time is mostly idle. The remedy is batching: process many requests’ decode steps together, so each pass over the weights serves many users. Throughput goes up sharply, and the main question becomes how to batch well. For general speed techniques see AI inference optimization.

The Problem with Static Batching

The naive approach collects a batch of requests, runs them together until all finish, then starts the next batch. Two things go wrong:

  1. Waiting for the slowest. Requests produce different numbers of tokens. A request that finishes after 20 tokens sits idle, its slot wasted, while a neighbor generates 800.
  2. Queueing delay. New requests must wait for the entire current batch to finish before they can start.

GPU utilization falls and latency rises, exactly the opposite of what you want.

Continuous Batching

Continuous batching (also called in-flight batching or iteration-level scheduling) makes scheduling decisions at every decode step rather than once per batch. After each step, finished sequences leave the batch immediately and waiting requests can join. The batch is a rolling set, not a fixed group.

Static:      [A A A A A A . . . .]   <- B, C finished early; slots idle
             [B B B . . . . . . .]
             [C C . . . . . . . .]

Continuous:  [A A A A A A A A A A]
             [B B B D D D D D . .]   <- D takes B's slot as soon as B ends
             [C C E E E E F F F F]   <- E, then F, reuse C's slot

The effect is higher utilization and lower waiting time. The cost is that scheduling is now a real piece of software: it must decide who runs next, handle prompts of different lengths, and manage memory dynamically. Which brings us to memory.

The KV Cache Memory Problem

Each active sequence needs a KV cache whose size depends on how long the sequence eventually becomes, and nobody knows that in advance. Early serving systems reserved a contiguous block sized for the maximum possible length for every request. That produces two kinds of waste:

  • Internal fragmentation: a request that uses 300 tokens of a 4,000-token reservation wastes the rest.
  • External fragmentation: free memory exists, but scattered in pieces too small for a new contiguous reservation.

Since the KV cache is usually what limits how many requests fit on a GPU, this waste directly caps throughput. A large share of the cache memory in such systems ended up unused, which is the motivation behind PagedAttention.

PagedAttention

PagedAttention borrows an idea from operating systems: virtual memory with paging. Instead of one contiguous region per sequence, the KV cache is divided into fixed-size blocks, each holding the keys and values for a small number of tokens. A per-sequence block table maps logical positions to physical blocks, which can sit anywhere in GPU memory.

Sequence 1 logical blocks:  [0] [1] [2]
Block table:                 7   2   9      -> physical blocks
Sequence 2 logical blocks:  [0] [1]
Block table:                 4   1

This brings several benefits:

  • Allocate on demand. A new block is allocated only when a sequence fills its current one, so waste is limited to the last partially filled block.
  • No external fragmentation. Any free block works for any sequence.
  • Sharing. Several sequences can point to the same physical blocks. This makes parallel sampling (multiple completions from one prompt) and beam search cheap, using copy-on-write when sequences diverge.
  • Higher concurrency. More free memory means more simultaneous requests, which means higher throughput.

The attention kernel is adapted to read keys and values through the block table rather than assuming one contiguous array. There is a small indirection overhead, which is far outweighed by the capacity gain.

Prefix Caching

If many requests share a prefix, such as a long system prompt or a fixed set of instructions, recomputing its KV cache for every request is wasteful. Because blocks are addressable, a server can hash blocks of prompt tokens and reuse cached blocks whose content matches. This is automatic prefix caching, and it cuts prefill work and time-to-first-token for workloads with repeated prompts. It is the server-side counterpart of the API feature discussed in prompt caching and production costs. To benefit, put stable content first in the prompt and variable content last.

Chunked Prefill

A long prompt’s prefill can monopolize the GPU for a while, stalling decode steps for everyone else and causing visible hiccups in streaming responses. Chunked prefill splits long prompts into pieces and interleaves them with decode work in the same batch. It smooths inter-token latency at a small cost to the long prompt’s own time-to-first-token. It is a tuning knob for balancing latency of existing streams against new arrivals.

Running vLLM

vLLM is an open-source serving engine built around these ideas. It exposes an OpenAI-compatible HTTP API, so existing client code can point at it with a changed base URL. A typical launch, with the model name as a placeholder for whatever checkpoint you are licensed to use:

pip install vllm

vllm serve your-org/your-model \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --port 8000

Then call it like any OpenAI-style endpoint:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-org/your-model",
    "messages": [{"role": "user", "content": "Explain paged memory in one paragraph."}],
    "max_tokens": 200,
    "stream": true
  }'

From Python with the official OpenAI client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="your-org/your-model",
    messages=[{"role": "user", "content": "Summarize continuous batching."}],
    max_tokens=150,
)
print(resp.choices[0].message.content)

For multi-GPU machines, tensor parallelism splits each layer across devices, which is needed when the model does not fit on one GPU:

vllm serve your-org/your-large-model --tensor-parallel-size 4

Flag names evolve between releases, so check the documentation for your installed version rather than copying options blindly.

Tuning the Key Knobs

  • --gpu-memory-utilization. The fraction of GPU memory the engine may use; after loading weights, most of the remainder becomes KV cache blocks. Too high risks out-of-memory errors from activation spikes or other processes; too low wastes capacity.
  • --max-model-len. The maximum context length. Setting it to what you actually need, rather than the model’s theoretical maximum, frees cache for more concurrent requests.
  • --max-num-seqs. The cap on concurrent sequences in a batch. Higher can raise throughput but also increases per-request latency.
  • Quantization. Running weights at lower precision (see quantization) shrinks the model and leaves more room for cache. Some engines can also quantize the KV cache itself, trading a little accuracy for capacity.
  • Speculative decoding. Can reduce latency for low-concurrency workloads; see speculative decoding.

Measuring What Matters

Pick metrics that match your product, and load test with realistic traffic, not a single repeated prompt:

  • Time to first token (TTFT): how long before the stream starts. Dominated by queueing and prefill.
  • Inter-token latency: the gap between streamed tokens. Matters for perceived fluency.
  • Throughput: tokens per second across all requests, which determines cost per token.
  • Tail latency: p95 and p99, because averages hide the slow requests users remember.

There is a fundamental tradeoff: pushing batch size up raises throughput and cost-efficiency but lengthens each user’s latency. The right operating point depends on whether you serve interactive chat, which needs low latency, or offline batch jobs, which should maximize throughput.

Alternatives and When Not to Use vLLM

vLLM is not the only option. Other engines such as TensorRT-LLM, SGLang and Hugging Face Text Generation Inference offer comparable techniques, with different strengths around hardware targets, structured output and kernel optimization. Compare them on your model and hardware instead of trusting generic claims.

You may not need a dedicated serving engine at all. For a single user or a prototype, a simple local runtime (running LLMs locally) is simpler. For low or spiky traffic, a managed API can be cheaper than keeping a GPU warm around the clock. Self-hosting pays off when volume is steady and high, when data cannot leave your network, or when you need control over the model and its behavior.

Operational Concerns

  • Health and autoscaling. GPUs are slow to start because weights must load. Keep warm capacity, use readiness probes that wait for the model to load, and scale on queue depth or concurrent requests rather than CPU.
  • Admission control. Set limits and return a clear error or queue position instead of letting the queue grow without bound.
  • Observability. Export metrics for queue length, KV cache usage and latency percentiles. Cache usage near 100 percent is a sign requests will be preempted or delayed.
  • Preemption. When memory runs out, an engine may pause or evict some sequences and recompute them later. This keeps the server alive but hurts latency, so treat frequent preemption as a signal to add capacity or lower limits.
  • Security. Do not expose the raw endpoint publicly without authentication and rate limiting.

Common Mistakes

  • Setting max-model-len to the model maximum “just in case,” wasting cache.
  • Benchmarking with identical short prompts, which hides prefill and fragmentation effects.
  • Optimizing throughput alone and shipping unacceptable tail latency.
  • Putting variable text at the start of the prompt and defeating prefix caching.
  • Pushing GPU memory utilization to the limit and getting out-of-memory crashes under traffic spikes.
  • Scaling on the wrong signal, such as CPU, for a GPU-bound service.

Key Takeaways

  • Decode is memory-bandwidth-bound, so batching many requests is the main lever for GPU efficiency.
  • Continuous batching schedules at every step, letting finished requests leave and new ones join instead of waiting for a whole batch.
  • PagedAttention stores the KV cache in fixed-size blocks mapped by a block table, removing fragmentation and enabling sharing and higher concurrency.
  • Prefix caching and chunked prefill improve time to first token and smooth streaming for real workloads.
  • vLLM exposes these through an OpenAI-compatible server; tune context length, memory utilization and concurrency to your traffic.
  • Measure TTFT, inter-token latency, throughput and tail latency with realistic load, and consider whether self-hosting beats a managed API for your volume.