Every time a chatbot writes a word, it has to look back at everything that came before it: your prompt, its own earlier words, the whole conversation. Doing that from scratch for each new word would waste enormous effort. So models keep a running notebook of what they already worked out about earlier text. That notebook is called the KV cache, and it quietly shapes how fast AI answers arrive, how many people one GPU can serve at once, and why very long conversations get expensive.
This explainer covers what the KV cache is, why it takes so much memory, and three research papers that attacked the problem from different angles. You do not need math to follow it. Where we quote numbers, they come from the papers, and we name the experiment each number belongs to.
What the KV cache is
Language models built on the transformer design (see our plain-English guide to transformers) work in a loop. They read text as tokens, which are short pieces of words (see what tokens are and how usage is billed). To pick the next token, the model uses a mechanism called attention. For every earlier token, attention holds two vectors, a list of numbers called a key and another called a value. The new token compares itself to all the keys to decide which earlier tokens matter, then blends their values into its answer.
The PagedAttention paper from UC Berkeley and collaborators describes the key point. The model can only generate tokens one by one, and each new token depends on the key and value vectors of all the tokens before it. Those vectors for old tokens do not change as the text grows, so serving systems store them instead of recomputing them. The paper writes that "the key and value vectors of existing tokens are often cached for generating future tokens, known as KV cache." At each new step, only one new key vector and one new value vector are computed. Everything older is read back from the cache.
A useful picture is a student taking notes in a lecture. Without notes, the student would re-listen to the whole lecture before writing each new sentence. With notes, the student reads the notes and adds one line. The cache is the notes. The price is the paper the notes are written on: they take up space, and they get longer with every word. For a serving-focused view of the same idea, see our earlier explainer on the KV cache as a memory bottleneck.
Two phases: reading your prompt, then writing the answer
The PagedAttention paper splits a request into two phases.
- The prompt phase. The model reads the whole prompt at once and fills the cache for every prompt token. Because all prompt tokens are known up front, this work can run in parallel, which suits a GPU well.
- The generation phase. The model writes the answer one token at a time. The paper says this phase cannot be parallelized because each step depends on the last. It "severely underutilizes GPU computation and becomes memory-bound," and it accounts for most of the latency of a single request.
Memory-bound means the chip spends its time waiting for data to arrive from memory, not doing arithmetic. At every step the chip must load the whole cache for the conversation so far. Noam Shazeer of Google made the same point in his 2019 paper: generating text step by step is slow because of the memory-bandwidth cost of repeatedly loading the large keys and values tensors.
That is the cost of a long conversation. A longer history means a bigger cache to load at every step, and a bigger cache to keep in memory for as long as the request lives.
How big the cache gets
The PagedAttention paper works through an example with the 13-billion-parameter OPT model. For one token, the cache takes 800 KB. The paper shows the arithmetic: 2 (one key and one value) times 5,120 (the size of the model's hidden state) times 40 (the number of layers) times 2 (the bytes used for each number in 16-bit format). OPT can handle sequences up to 2,048 tokens, so one request can need as much as 1.6 GB of cache.
The paper also gives a snapshot of a GPU serving that model on an NVIDIA A100 with 40 GB of memory. About 65 percent of the memory holds the model weights, which stay fixed. Roughly 30 percent holds the dynamic states of requests, which are the keys and values of the KV cache. Because the weights are a fixed cost, the paper concludes that how the cache is managed decides the maximum batch size, meaning how many requests the GPU can serve together. Even if all free memory went to the cache, the authors note, only a few tens of requests would fit.
That ties the cache directly to cost. If each request needs less cache memory, more requests fit on one GPU, and the cost per request falls. The paper cites an estimate that processing an LLM request can be 10 times more expensive than a traditional keyword query, which is why serving teams care about squeezing out waste.
Attack one: stop wasting cache memory (PagedAttention and vLLM)
The first problem the PagedAttention authors found is plain waste. Older serving systems kept each request's cache in one contiguous block of memory and reserved room for the maximum possible length, for example 2,048 tokens, even when the real answer was short. The paper's profiling found that only 20.4 percent to 38.2 percent of the cache memory in those existing systems stored actual token states. The rest was reserved but unused, or lost to gaps between blocks.
Their fix borrows from operating systems. PagedAttention splits each request's cache into fixed-size blocks, like pages in virtual memory, and allows the blocks to sit anywhere in memory. A table maps the request's logical blocks to physical ones. New blocks are handed out only when the old ones fill up, so waste is limited to the last block of each request. The authors built a serving engine on top of it called vLLM, with a default block size of 16 tokens. Our deep dive on the math of PagedAttention covers the block table in detail.
The headline result, from the paper's evaluation on OPT and LLaMA models using A100 GPUs, is that vLLM improves the throughput of popular LLMs by 2 to 4 times at the same level of latency compared with the systems they tested against, FasterTransformer and Orca. The gap grows with longer sequences, larger models, and more complex decoding methods. The authors report that this comes without affecting model accuracy at all, since the model computes the same thing and only the memory layout changes.
Paged blocks also make sharing possible. When several outputs come from the same prompt, they can point at the same stored prompt blocks, with a copy made only when one output needs to write something different. The paper measured memory saved by sharing on the OPT-13B model with the Alpaca dataset:
- Parallel sampling (several answers to one prompt): 6.1 percent to 9.8 percent of cache memory saved.
- Beam search (a method that keeps several candidate answers alive): 37.6 percent to 55.2 percent saved.
For the ShareGPT dataset, which has longer prompts, the same experiments saved 16.2 percent to 30.5 percent for parallel sampling and 44.3 percent to 66.3 percent for beam search.
The same idea covers a shared prefix. Many requests start with the same long instructions, such as a system prompt with examples. A provider can store the cache for that prefix once and let every request reuse it. In the paper's translation test on LLaMA-13B, sharing a prefix with one example gave 1.67 times the throughput of the Orca (Oracle) baseline, and a prefix with five examples gave 3.58 times. Those numbers compare against one specific baseline, an idealized version of Orca that knows each output length in advance.
The paper is open about costs. Reading paged memory made the attention kernel itself 20 percent to 26 percent slower than the highly optimized FasterTransformer version. The authors say they believe the overhead is small because it only affects the attention operator, not the other operators in the model, and vLLM still beats FasterTransformer in end-to-end tests.
Attack two: make the cache smaller (multi-query attention)
PagedAttention manages the cache better. A second line of work makes it smaller. A transformer layer runs several attention "heads" in parallel, and in the standard design every head keeps its own keys and values. Shazeer's paper proposes multi-query attention, where all heads share one set of keys and values while each head keeps its own queries. That shrinks the cache and the data loaded at each step.
Shazeer tested this on a translation model with 211 million parameters, using the WMT 2014 English to German task, where the baseline had 8 heads. The speed numbers are in TPUv2 microseconds per output token, for a sequence length of 128:
- Multi-head baseline, decoder: 46 microseconds per token.
- Multi-query, decoder: 3.8 microseconds per token.
- Multi-head baseline with beam search, decoder: 203 microseconds per token.
- Multi-query with beam search, decoder: 32 microseconds per token.
Quality moved only a little. On the development set the baseline scored 26.7 BLEU (a standard translation score where higher is better) and multi-query scored 26.5. On the test set with beam search of width 4, multi-query scored 28.5 against 28.4 for the baseline. That 0.1 BLEU gap favors multi-query, although the dev-set BLEU of 26.5 against 26.7 and the beam-1 test BLEU of 27.5 against 27.7 favor the baseline. On the Billion-Word language modeling benchmark, multi-query's perplexity (lower is better) was 30.2 against 29.9 for the baseline, so slightly worse for multi-query. Shazeer summarizes it as models that are "much faster to decode" with "only minor quality degradation." Training speed was about the same, 13.0 against 13.2 microseconds per token.
One caution: this is a 211 million parameter model on translation, not a modern chat model. The next paper tests T5 XXL, a larger model.
Attack three: keep the quality (grouped-query attention)
A Google Research paper on GQA points out two problems with multi-query attention. It can cost quality and cause training instability, and many popular models were already trained with standard multi-head attention, so retraining from scratch is costly. The authors offer two ideas.
The first is a recipe to convert an existing multi-head model into a multi-query one by "uptraining." The key and value projections of all heads are averaged into one, and the model then trains a little more, using 5 percent of its original pre-training compute. The second is grouped-query attention. Query heads are split into groups, and each group shares one set of keys and values. One group is the same as multi-query attention. As many groups as heads is the same as the standard design. Everything in between is a dial between cache size and quality. Our guide to the math of grouped-query and multi-query attention covers the arithmetic.
The paper tested this on the T5 XXL model across summarization, translation, and question answering tasks. These are the inference time per sample on TPUv4 chips and the average score over all tasks, as the paper reports them (four rows):
- Multi-head T5-Large: 0.37 seconds, average 46.0.
- Multi-head T5-XXL: 1.51 seconds, average 47.2.
- Multi-query T5-XXL, uptrained: 0.24 seconds, average 46.6.
- GQA with 8 groups, T5-XXL, uptrained: 0.28 seconds, average 47.1.
So the 8-group version took 0.28 seconds per sample, against 0.24 for multi-query and 1.51 for the full multi-head model, while its average score landed 0.1 below that model. The authors chose 8 groups as a middle ground. They also found that mean-pooling the heads worked better than picking one head or starting from scratch, and that GQA worked reasonably right after conversion while multi-query needed the extra training to be useful. On the single WMT translation column, multi-query scored 28.5 BLEU and multi-head XXL scored 28.4, so on that one task the smaller cache scored 0.1 higher than the full model.
The authors list limits. They evaluated only encoder-decoder models, not decoder-only models. The paper notes that decoder-only models are now extremely popular, and expects GQA to have a stronger advantage over multi-query attention there, but did not test it. They also note their summarization metric, ROUGE, is flawed, so they cannot be certain the trade-offs are exactly right. In addition, they did not compare their GQA model against one trained from scratch. And in an appendix they report that multi-query attention trained from scratch showed frequent loss spikes and diverged when fine-tuned on long-input tasks, while the uptrained GQA models appeared stable.
What this means in practice
A few plain takeaways follow from these papers, each tied to what they measured.
- Long contexts cost memory. The cache grows with every token in the conversation, and the OPT-13B example shows that one long request can take gigabytes. A bigger context window is not free, because the cache has to fit somewhere.
- Serving speed is often about memory, not math. All three papers describe generation as memory-bound, so tricks that move less data or waste less memory can matter as much as faster chips.
- Shared prefixes can be cheap to reuse. Where many requests start with the same instructions, storing that prefix's cache once saves recomputation, as the paper's prefix tests show.
- Model design bakes in cache decisions. The choice between multi-head, grouped-query, and multi-query attention sets how big the cache is for every user of that model, and it is fixed when the model is trained, unless a team uptrains it as the GQA paper describes.
These papers are from 2019 and 2023, so treat their numbers as measurements of the systems, models and hardware they tested, not of any current product.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al., SOSP 2023
- Fast Transformer Decoding: One Write-Head is All You Need, Noam Shazeer, 2019
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Ainslie et al., Google Research
- vLLM source code, linked from the PagedAttention paper
