Attention Variants and KV-Cache Compression
During autoregressive decoding, each new token attends to the keys and values of all earlier tokens; variants such as sliding-window attention limit that range. Storing those keys and values is the KV cache. It avoids recomputing them, but cache memory grows linearly with sequence length and with the number of sequences decoded at once (batch size): in the 32-layer example below, one sequence of 4096 tokens needs 512 MiB of cache, and at a 128K context (32 times 4096) it needs 16 GiB.
With long contexts or many sequences decoding at once, the bytes read from the cache each step can match or exceed the bytes read for the weights; with a short context and a single sequence, weight reads still dominate.
There are three ways to shrink the cache: let several query heads share one set of keys and values (fewer KV heads), store something narrower for each token (a narrower cached representation), or keep fewer tokens and store them at lower precision (fewer or lower-precision cached tokens). The first two are fixed at training time; the third works on an existing model's cache.
Open full-size imageRead upward from the blue query heads. In the middle, each pair of queries shares one key/value pair; on the right, all queries share one pair. The query count stays fixed. The cache saving comes from storing fewer keys and values, with other dimensions held constant.
MHA and GQA
The size of the cache depends on how many KV heads each layer stores keys and values for. Let a model have layers, sequence length , query heads, KV heads, head dimension , and bytes per cached element. The three attention types differ only in :
- Multi-head attention (MHA) normally uses : each query head has its own KV head.
- Multi-query attention (MQA) uses one KV head shared by all query heads.
- Grouped-query attention (GQA) lies between them: several query heads share each KV head.
Ignoring allocator and metadata overhead, the KV cache for one sequence is approximately
In words: for every layer and every token, each KV head stores one key and one value of numbers each, which is where the factor of two comes from; multiply by the number of layers, the number of tokens, and the bytes per number.
For a hypothetical , , , , , and two-byte elements, work through it in that order:
With MHA, each of the 32 query heads gets its own KV head, so goes from 8 to 32 and the cache quadruples to 2 GiB. In other words, holding the other dimensions fixed, moving from MHA to GQA shrinks the cache in proportion to , here , a quarter. The cache also grows linearly with concurrency: four equal-length GQA sequences use 2 GiB as well, before overhead. These figures cover the cache only; the weights take memory of their own.
GQA is an architectural choice made during training, not a lossless switch applied to arbitrary weights. The recipe in the GQA paper merges an existing multi-head model's key and value heads into groups, then continues training with about 5% of the original pre-training compute. The resulting GQA model comes close to multi-head quality at a speed close to MQA.
Every new token adds one key and one value per K/V head in every layer. Choose how the query heads share K/V heads, then step through decoding.
32 query heads → 8 K/V heads
token 12Decoding token 12: append one row (128 KiB over all layers), then the new query reads all 12 rows: 1.5 MiB for this sequence.
one row = 2 (K and V) × 32 layers × 8 K/V heads × 128 × 2 B = 128 KiB
At a real context length
- KV cache
- 512 MiB
- 4,096 × 1 × 128 KiB
- Read per decoding step
- 512 MiB
- per sequence, for every new token
- Compared with MHA
- 25%
- 8 / 32 K/V heads
With 1 concurrent sequence(s), the GQA cache outgrows those weights (14.9 GiB) from 122,071 tokens of context.
Shape of the note's hypothetical example: 32 layers, 32 query heads, head dimension 128. The 8-billion-parameter weight line is an assumed size. Allocator overhead and activations are not counted.
When Is Reuse Exact?
A cache can be reused unchanged only if earlier tokens' keys and values stay the same when a new token arrives. A causal model meets that condition: appending a token does not change previous token representations, and each layer computes only the new positions. Two requests that start the same way can therefore share that stretch of cache, which is where a runtime's prefix cache comes from. If the tokenized prefix, the weights (including adapters) or the position indices differ anywhere, or an earlier token is edited, every cached state from that point on is invalid.
Tree speculation extends this state-management problem to candidate branches: isolate them and retain KV and compressed states along the accepted path.
Multi-head latent attention
GQA saves cache by storing fewer heads. DeepSeek-V2 introduced multi-head latent attention (MLA), which takes another route: instead of caching full per-head key and value tensors, the model learns a lower-dimensional latent representation, caches only that short vector for each token, and projects it back into per-head keys and values when needed. Parts of the key path used for rotary position information (RoPE) remain separate, as a small extra piece, because the position-dependent rotation cannot simply be folded into a fixed projection.
A simplified cache-width comparison is:
where is the compressed latent width and is the decoupled rotary-key width. Put differently, the MHA/GQA width grows with the number of KV heads, while MLA stores one latent vector plus one position key, independent of the head count.
With the configuration given in the DeepSeek-V2 paper: 60 layers, 128 attention heads of 128 dimensions each, latent width , and rotary-key width . The table assumes two bytes per number:
MLA stores about one fifty-seventh as many numbers per token as MHA of the same shape (). Its 576 is even below the 2,048 per layer of the 8-KV-head GQA example above; the paper puts it as a cache equal to GQA with only 2.25 groups ().
Like GQA, MLA is fixed at training time: the compression and reconstruction projections are learned, and the paper's results belong to models trained with that architecture.
When running a model, you can turn length, precision and prefix caching
The attention structure is fixed at training time; whoever runs the model cannot change it. What you can change are the formula's other two numbers: how many tokens the cache keeps () and how many bytes each value takes (), plus whether an identical opening is computed a second time.
With the GQA example above: 128 KiB per token makes 4 GiB at a 32K context; q8_0 brings that to about 2 GiB and q4_0 to about 1 GiB. How a whole machine's budget splits between weights and cache is in single-device deployment.
Compressing an existing cache
Another family of research methods leaves the trained architecture alone and deletes tokens from the cache. SnapKV scores earlier positions with the attention of an observation window at the end of the prompt and keeps only the high-scoring keys and values. PyramidKV gives layers different cache budgets according to how their attention is distributed. The memory they save comes from storing fewer tokens, at the risk of deleting a small but decisive passage.
What to measure
For a concrete deployment, record:
- exact model, attention architecture, and cache dtype;
- prompt length, generated length, and concurrent sequences;
- peak GPU and host memory;
- prefill time, time to first token, and decode throughput;
- task success on long-context cases that require information from different positions;
- the behavior after cache eviction, quantization, or CPU offload.
Memory fit and retained answer quality need separate tests: a model that advertises a long context window may not fit the available hardware, while a cache-saving method that fits may still fail the task.