KV Cache Explained: Why Long Conversations Get Expensive

If you have ever noticed that the fifth message in a long chat takes longer to get a response than the first, or that a self-hosted model that handled a short prompt fine suddenly runs out of GPU memory once the conversation gets long, you have run into the KV cache. It is the single biggest reason serving long-context conversations is expensive, and it has nothing to do with the size of the model’s weights.

Why Generation Needs a Cache at All

A transformer generates one token at a time, and each new token’s attention computation needs the key and value vectors of every token that came before it. Without caching, generating token N would mean recomputing the key and value projections for all N-1 previous tokens, every single time — an O(n²) amount of redundant work over the course of a response.

The KV cache eliminates that redundancy: once a token’s key and value vectors are computed, they are stored, and every subsequent step just reuses them and computes the K/V pair for the one new token. This turns per-token generation cost from growing with the whole history into a small, roughly constant increment — at the cost of memory that holds all those stored vectors.

Step 1: compute K,V for token 1                     → cache: [K1,V1]
Step 2: compute K,V for token 2, attend to [K1,V1]  → cache: [K1,V1,K2,V2]
Step 3: compute K,V for token 3, attend to K1..K2   → cache: [K1,V1,K2,V2,K3,V3]
...

Without this cache, an interactive chat model would recompute the entire prompt’s key/value projections on every single output token, which is untenable for anything beyond a few hundred tokens of context.

Sizing the Cache

The memory the KV cache occupies is a function of model shape and sequence length, not of how “smart” the model is. For a standard multi-head attention (MHA) transformer:

kv_cache_bytes = 2 (K and V)
                × num_layers
                × num_heads
                × head_dim
                × seq_len
                × batch_size
                × bytes_per_element

A concrete example: a 32-layer model with 32 attention heads of dimension 128, running at 16-bit precision, for a single 8,000-token conversation:

num_layers = 32
num_heads = 32
head_dim = 128
seq_len = 8_000
batch_size = 1
bytes_per_element = 2  # fp16/bf16

kv_cache_bytes = 2 * num_layers * num_heads * head_dim * seq_len * batch_size * bytes_per_element
print(kv_cache_bytes / (1024 ** 3), "GiB")

That single example lands around 4 GiB just for the cache of one 8k-token conversation — on top of whatever memory the model’s own weights occupy. Serve 20 such conversations concurrently and the cache alone can dwarf the weights. This is precisely why “how many concurrent users can this GPU handle” is usually a KV-cache capacity question, not a compute question, once context lengths get long.

Multi-Query and Grouped-Query Attention

The num_heads term in that formula is the main lever architects have pulled to control cache size. Standard MHA gives every attention head its own key and value projections. Multi-query attention (MQA) instead shares a single key/value head across all query heads, and grouped-query attention (GQA) is the middle ground: query heads are split into a handful of groups, and each group shares one KV head.

Attention variant	KV heads	Relative cache size	Typical quality impact
Multi-head attention (MHA)	Equal to number of query heads	1x (baseline)	Best quality per parameter
Grouped-query attention (GQA)	A small fixed number of groups (e.g. 8)	~4-8x smaller	Small, usually acceptable quality drop
Multi-query attention (MQA)	1	Largest reduction (proportional to head count)	Larger quality drop on some tasks

GQA in particular has become close to a default choice for new open-weight models, because it captures most of the memory savings of MQA while staying much closer to MHA quality. If you are comparing two similarly-sized open models and one advertises GQA, expect it to support meaningfully longer context or higher concurrency on the same hardware.

Paged Attention: Treating the Cache Like Virtual Memory

Even with GQA, naively allocating a contiguous block of memory sized for the maximum possible sequence length per request wastes enormous amounts of memory, because most requests don’t use their full context window and the wasted space (internal fragmentation) can’t be reused by other requests.

Paged attention, popularized by the vLLM serving engine, borrows the idea of paging from operating systems: the KV cache is split into fixed-size blocks, and a request’s cache is a list of block pointers rather than one contiguous allocation. Blocks are allocated on demand as generation proceeds and freed immediately when a request finishes, and blocks can even be shared between requests that have an identical prefix (for example, the same system prompt).

Without paging: reserve worst-case contiguous memory per request → heavy fragmentation
With paging:    allocate small fixed-size blocks on demand        → near-zero waste,
                                                                     shareable prefixes

This is one of the reasons a self-hosted vLLM or TGI deployment can sustain dramatically higher concurrent throughput than a naive HuggingFace generate() loop on the same GPU: the memory manager, not just the raw compute, is doing real work.

Quantized and Offloaded KV Caches

For deployments that are memory-bound rather than compute-bound, two further techniques trade a bit of accuracy or latency for cache capacity:

• KV cache quantization stores keys and values in 8-bit or even 4-bit precision instead of 16-bit, roughly halving or quartering cache size. Because K/V vectors go through a softmax and weighted sum rather than a chain of matrix multiplications, they tend to tolerate quantization somewhat better than weights do, though degradation is still visible at very low bit-widths on long contexts.
• CPU/NVMe offloading moves the KV cache for older, less-recently-attended tokens out of GPU memory and streams it back in when needed. This trades a PCIe round-trip in latency for the ability to hold far more concurrent context than GPU memory alone would allow — useful for batch or asynchronous workloads where latency variance is tolerable, less so for a live chat UI.

# Example vLLM server flags relevant to KV cache behavior
--kv-cache-dtype fp8
--gpu-memory-utilization 0.90
--max-num-seqs 64
--enable-prefix-caching

Prefix Sharing: The Other Half of the Picture

A closely related optimization is reusing KV cache entries across requests that share an identical prefix — the same system prompt, the same set of tool definitions, the same few-shot examples. Because the key/value vectors for a given prefix are deterministic given the same weights and input, they don’t need to be recomputed per request; they can be computed once and referenced by every request that starts with that exact prefix. This is the serving-side mechanism that makes provider-side prompt caching features possible, and it rewards structuring prompts with a stable, identical prefix across calls rather than interleaving variable content near the start.

Practical Guidance

For a small, self-hosted deployment (the kind of single-VPS-plus-GPU setup many teams run), the KV cache is usually the first thing to blow your memory budget as you raise context length or concurrency, well before you’d expect the model weights to be the bottleneck. A few rules of thumb:

• If you’re choosing between two open-weight models of similar quality, prefer the one using GQA — it will support longer context or more concurrent users on the same GPU.
• Use a serving engine with paged attention and prefix caching (vLLM, TGI, and similar) rather than a bare transformers.generate() loop for anything beyond single-user local testing.
• If you’re memory-constrained, try KV cache quantization before reaching for a smaller model — it often preserves more quality per byte saved.
• Cap max_tokens and conversation history length deliberately; unbounded chat history is the most common cause of a KV cache silently growing until a request fails.

None of this shows up as a line item anywhere obvious. A model that “suddenly” runs out of memory after handling a dozen long conversations fine for weeks isn’t degrading — it’s hitting a capacity wall that was always there, just never exercised until concurrency and context length both happened to be high at the same time. Treating KV cache capacity as an explicit, monitored resource, the same way you’d monitor disk space or connection pool exhaustion, turns that surprise into a predictable, budgeted constraint.

Key Takeaways

• The KV cache stores every previous token’s key and value vectors so generation doesn’t recompute them at every step; it is what makes autoregressive decoding tractable at all.
• Its memory footprint scales with layers, heads, head dimension, sequence length, and batch size — not with model “intelligence” — and for long conversations it can dwarf the model’s own weights.
• Grouped-query and multi-query attention reduce cache size by sharing key/value heads across query heads, at a small to moderate quality cost.
• Paged attention manages the cache like OS virtual memory, allocating fixed-size blocks on demand to eliminate fragmentation and enable sharing identical prefixes across requests.
• KV cache quantization and CPU/NVMe offloading trade some accuracy or latency for more effective capacity when memory, not compute, is the bottleneck.
• Prefix sharing across requests with an identical stable prefix is the serving-side mechanism behind provider-side prompt caching.
• In self-hosted deployments, the KV cache is usually the first resource to run out as context length or concurrency grows, so plan capacity around it explicitly rather than around raw GPU FLOPs.