The KV cache
What the key-value cache stores, why it dominates serving memory, and the techniques that shrink or manage it: paging, grouped-query attention and prefix reuse.
What it stores
Self-attention compares each new token against all earlier tokens. The keys and values computed for earlier tokens do not change, so the model keeps them instead of recomputing them at every step. That store is the KV cache.
Diagram: my own.
Prefill fills the cache for the prompt. Decode reads it at every step and appends one token’s worth. It is a trade of memory for compute, and the memory is paid per request.
Why it matters
A common formula for its size is layers times KV heads times 2 (keys and values) times sequence length times head dimension times bytes per element3. It grows linearly with context length and with the number of concurrent requests. On a GPU the model’s weights are fixed, and what is left over goes to the KV caches of all active requests, so the cache limits how many requests you can batch.
The PagedAttention paper makes this point: each request’s KV cache is large and grows and shrinks dynamically, so inefficient management wastes memory through fragmentation and duplication and limits batch size1.
Paging the cache
PagedAttention borrows virtual memory from operating systems. It splits each request’s KV cache into fixed-size blocks that need not be contiguous, so blocks are allocated as the sequence grows1. Blocks can also be shared across sequences, including across requests with a common prefix1. The vLLM engine built on it reports near-zero cache waste and 2 to 4 times the throughput of the prior systems it compared against, at the same latency1. Treat the multipliers as the authors’ benchmark results, not a promise for your workload.
Diagram: my own.
Making the cache smaller
Fewer K/V heads. In multi-head attention every query head has its own keys and values. Multi-query attention shares one K/V head across all query heads, and grouped-query attention shares one per group, so the cache scales with the number of distinct K/V heads rather than query heads2. This is a model design choice made at training time, so you benefit by picking models that use it.
Diagram: my own.
Lower precision. Some engines can store the cache in fewer bits. This reduces memory at some risk to accuracy (my own note; check your engine’s docs and test).
Shorter contexts. The cheapest cache is the one you do not fill. Trim what you send (Context window as a budget, Long-context trade-offs).
Reusing the cache across requests
If two requests start with the same tokens, their KV entries for that prefix are identical, so a server can keep and reuse them. SGLang’s RadixAttention is one design for this: it reuses the KV cache across generation calls that share prefixes4. Hosted APIs expose the same idea as prompt caching. The saving only applies to an exact shared prefix, which is why prompt layout matters.
Practical takeaways
- Budget memory as weights plus per-request cache, not weights alone.
- When you see out-of-memory errors or small batches on long prompts, suspect the cache.
- Prefer models with grouped-query attention if you self-host.
- Put stable content first so prefixes can be reused.
- Remember that long agent runs keep growing their cache and their bill (Latency and cost).
Sources
- [1]Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al. · 2023PARAPHRASED
- [2]Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — Brandon et al. · 2024BACKGROUND SECTION ON MQA AND GQA
- [3]Multi-Query and Grouped-Query Attention: Shrinking the KV Cache — General Compute · 2026CACHE-SIZE FORMULA, AS SUMMARISED BY A THIRD PARTY
- [4]SGLang: Efficient Execution of Structured Language Model Programs — Zheng et al. · 2024PARAPHRASED