Skip to main content

Attention Variants and KV-Cache Compression

During autoregressive decoding, each new token attends to the keys and values of all earlier tokens; variants such as sliding-window attention limit that range. Storing those keys and values is the KV cache. It avoids recomputing them, but cache memory grows linearly with sequence length and with the number of sequences decoded at once (batch size): in the 32-layer example below, one sequence of 4096 tokens needs 512 MiB of cache, and at a 128K context (32 times 4096) it needs 16 GiB.

With long contexts or many sequences decoding at once, the bytes read from the cache each step can match or exceed the bytes read for the weights; with a short context and a single sequence, weight reads still dominate.

There are three ways to shrink the cache: let several query heads share one set of keys and values (fewer KV heads), store something narrower for each token (a narrower cached representation), or keep fewer tokens and store them at lower precision (fewer or lower-precision cached tokens). The first two are fixed at training time; the third works on an existing model's cache.

MHA, GQA and MQA retain eight query heads while using eight, four and one shared key/value heads.Open full-size image

Read upward from the blue query heads. In the middle, each pair of queries shares one key/value pair; on the right, all queries share one pair. The query count stays fixed. The cache saving comes from storing fewer keys and values, with other dimensions held constant.

MHA and GQA​

The size of the cache depends on how many KV heads each layer stores keys and values for. Let a model have LL layers, sequence length SS, nhn_h query heads, nkvn_{kv} KV heads, head dimension dhd_h, and bb bytes per cached element. The three attention types differ only in nkvn_{kv}:

  • Multi-head attention (MHA) normally uses nkv=nhn_{kv}=n_h: each query head has its own KV head.
  • Multi-query attention (MQA) uses one KV head shared by all query heads.
  • Grouped-query attention (GQA) lies between them: several query heads share each KV head.

Ignoring allocator and metadata overhead, the KV cache for one sequence is approximately

2LSnkvdhb.2L S n_{kv} d_h b.

In words: for every layer and every token, each KV head stores one key and one value of dhd_h numbers each, which is where the factor of two comes from; multiply by the number of layers, the number of tokens, and the bytes per number.

For a hypothetical L=32L=32, S=4096S=4096, nh=32n_h=32, nkv=8n_{kv}=8, dh=128d_h=128, and two-byte elements, work through it in that order:

What is being computedExpressionResult
numbers stored per token per layer2×8×1282\times 8\times 1282,048
numbers per token across 32 layers2,048×322{,}048\times 3265,536
bytes per token65,536×265{,}536\times 2131,072 bytes, or 128 KiB
total for 4096 tokens131,072×4096131{,}072\times 4096536,870,912536{,}870{,}912 bytes, or 512 MiB

With MHA, each of the 32 query heads gets its own KV head, so nkvn_{kv} goes from 8 to 32 and the cache quadruples to 2 GiB. In other words, holding the other dimensions fixed, moving from MHA to GQA shrinks the cache in proportion to nkv/nhn_{kv}/n_h, here 8/328/32, a quarter. The cache also grows linearly with concurrency: four equal-length GQA sequences use 2 GiB as well, before overhead. These figures cover the cache only; the weights take memory of their own.

GQA is an architectural choice made during training, not a lossless switch applied to arbitrary weights. The recipe in the GQA paper merges an existing multi-head model's key and value heads into groups, then continues training with about 5% of the original pre-training compute. The resulting GQA model comes close to multi-head quality at a speed close to MQA.

What the KV cache stores while a model writes

Every new token adds one key and one value per K/V head in every layer. Choose how the query heads share K/V heads, then step through decoding.

Attention
Cache element

32 query heads → 8 K/V heads

123456789101112

token 12Decoding token 12: append one row (128 KiB over all layers), then the new query reads all 12 rows: 1.5 MiB for this sequence.

one row = 2 (K and V) × 32 layers × 8 K/V heads × 128 × 2 B = 128 KiB

At a real context length

Concurrent sequences
KV cache
512 MiB
4,096 × 1 × 128 KiB
Read per decoding step
512 MiB
per sequence, for every new token
Compared with MHA
25%
8 / 32 K/V heads
0 B32 GiB64 GiB032K64K96K128KMHA · 64 GiBGQA · 16 GiBMQA · 2 GiB

With 1 concurrent sequence(s), the GQA cache outgrows those weights (14.9 GiB) from 122,071 tokens of context.

Shape of the note's hypothetical example: 32 layers, 32 query heads, head dimension 128. The 8-billion-parameter weight line is an assumed size. Allocator overhead and activations are not counted.

When Is Reuse Exact?​

A cache can be reused unchanged only if earlier tokens' keys and values stay the same when a new token arrives. A causal model meets that condition: appending a token does not change previous token representations, and each layer computes only the new positions. Two requests that start the same way can therefore share that stretch of cache, which is where a runtime's prefix cache comes from. If the tokenized prefix, the weights (including adapters) or the position indices differ anywhere, or an earlier token is edited, every cached state from that point on is invalid.

Tree speculation extends this state-management problem to candidate branches: isolate them and retain KV and compressed states along the accepted path.

Multi-head latent attention​

GQA saves cache by storing fewer heads. DeepSeek-V2 introduced multi-head latent attention (MLA), which takes another route: instead of caching full per-head key and value tensors, the model learns a lower-dimensional latent representation, caches only that short vector for each token, and projects it back into per-head keys and values when needed. Parts of the key path used for rotary position information (RoPE) remain separate, as a small extra piece, because the position-dependent rotation cannot simply be folded into a fixed projection.

A simplified cache-width comparison is:

MHA/GQA width per token per layer≈2nkvdh,MLA width per token per layer≈dc+dR,\begin{aligned} \text{MHA/GQA width per token per layer} &\approx 2n_{kv}d_h,\\ \text{MLA width per token per layer} &\approx d_c+d_R, \end{aligned}

where dcd_c is the compressed latent width and dRd_R is the decoupled rotary-key width. Put differently, the MHA/GQA width grows with the number of KV heads, while MLA stores one latent vector plus one position key, independent of the head count.

With the configuration given in the DeepSeek-V2 paper: 60 layers, 128 attention heads of 128 dimensions each, latent width dc=512d_c=512, and rotary-key width dR=64d_R=64. The table assumes two bytes per number:

AttentionNumbers per token per layerPer token, 60 layers4096 tokens
MHA with the same heads2×128×128=32,7682\times 128\times 128=32{,}7683.75 MiB15 GiB
MLA512+64=576512+64=57667.5 KiB270 MiB

MLA stores about one fifty-seventh as many numbers per token as MHA of the same shape (32,768÷576≈5732{,}768\div 576\approx 57). Its 576 is even below the 2,048 per layer of the 8-KV-head GQA example above; the paper puts it as a cache equal to GQA with only 2.25 groups (2×2.25×128=5762\times 2.25\times 128=576).

Like GQA, MLA is fixed at training time: the compression and reconstruction projections are learned, and the paper's results belong to models trained with that architecture.

When running a model, you can turn length, precision and prefix caching​

The attention structure is fixed at training time; whoever runs the model cannot change it. What you can change are the formula's other two numbers: how many tokens the cache keeps (SS) and how many bytes each value takes (bb), plus whether an identical opening is computed a second time.

KnobHow to set itCost
Context lengthOllama defaults to 4096 tokens; change it with OLLAMA_CONTEXT_LENGTH or num_ctx in the request. llama.cpp uses -c, where 0 means the length stored in the model. mlx-lm switches to a fixed-size rotating cache with --max-kv-sizetoo short truncates the input; the smaller the rotating cache, the less memory and the lower the quality
Cache precisionOllama's OLLAMA_KV_CACHE_TYPE: q8_0 is about half the default f16, q4_0 about a quarter; it needs Flash Attention and applies to all models. llama.cpp uses -ctk and -ctv, default f16, with q8_0, q4_0 and others available. vLLM sets FP8 with kv_cache_dtypethe lower the precision, the more attention scores drift; vLLM's documentation notes that sliding-window and similar layers are more sensitive to cache quantization
Prefix cachingllama.cpp's server has --cache-prompt on by default; vLLM uses enable_prefix_caching=Truesaves prefill only, not token-by-token generation; a different opening cannot hit

With the GQA example above: 128 KiB per token makes 4 GiB at a 32K context; q8_0 brings that to about 2 GiB and q4_0 to about 1 GiB. How a whole machine's budget splits between weights and cache is in single-device deployment.

Compressing an existing cache​

Another family of research methods leaves the trained architecture alone and deletes tokens from the cache. SnapKV scores earlier positions with the attention of an observation window at the end of the prompt and keeps only the high-scoring keys and values. PyramidKV gives layers different cache budgets according to how their attention is distributed. The memory they save comes from storing fewer tokens, at the risk of deleting a small but decisive passage.

What to measure​

For a concrete deployment, record:

  1. exact model, attention architecture, and cache dtype;
  2. prompt length, generated length, and concurrent sequences;
  3. peak GPU and host memory;
  4. prefill time, time to first token, and decode throughput;
  5. task success on long-context cases that require information from different positions;
  6. the behavior after cache eviction, quantization, or CPU offload.

Memory fit and retained answer quality need separate tests: a model that advertises a long context window may not fit the available hardware, while a cache-saving method that fits may still fail the task.

Explore connectionsOpen network