Attention Evolution & KV Cache Compression

In autoregressive Large Language Model (LLM) inference, the Self-Attention decoding stage is strictly memory-bandwidth bound. Under ultra-long context generation, Key-Value (KV) Cache memory scales linearly with sequence length, rapidly surpassing the memory footprint of the static model weights themselves.

---

1. Architectural Evolution: MHA to GQA and MLA

---

2. Memory & Computational Complexity Comparison

For hidden dimension $d$, head count $n_h$, head dimension $d_h = d / n_h$, layer count $L$, and sequence length $S$:

2.1 Multi-Head Attention (MHA)

Standard MHA allocates independent Key and Value heads for each Query head:

$$ext{KV Cache Size (MHA)} = 2 imes L imes n_h imes d_h imes S imes b_{ ext{elem}}$$

• Bottleneck: With $n_h = 128$ and $S = 128 ext{k}$, a single request KV Cache exceeds 50 GB, causing rapid VRAM exhaustion.

2.2 Grouped-Query Attention (GQA)

GQA groups $n_h$ Query heads into $n_{kv}$ shared Key/Value heads (typically $n_{kv} = 8$):

$$ext{KV Cache Size (GQA)} = 2 imes L imes n_{kv} imes d_h imes S imes b_{ ext{elem}}$$

• Trade-off: Achieves a fixed compression ratio of $n_h / n_{kv}$ (typically $4 imes$–$8 imes$), but head sharing incurs minor expressiveness degradation in complex multi-document retrieval.

---

3. DeepSeek MLA (Multi-Head Latent Attention) Architecture

DeepSeek MLA introduces low-rank joint compression, achieving over 93% KV Cache reduction while preserving full multi-head expressive power during inference.

3.1 Low-Rank Projection & Latent Vector Caching

Instead of caching individual multi-head Keys and Values, MLA projects the hidden state $h_t$ into a shared low-rank latent vector $c_t^{(KV)} \in \mathbb{R}^{d_c}$ where $d_c \ll n_h imes d_h$:

$$c_t^{(KV)} = W_{DKV} h_t, \quad [k_{t,1}^{(C)}, \dots, k_{t,n_h}^{(C)}] = W_{UK} c_t^{(KV)}, \quad [v_{t,1}^{(C)}, \dots, v_{t,n_h}^{(C)}] = W_{UV} c_t^{(KV)}$$

• Inference Invariant: In VRAM, only the single latent vector $c_t^{(KV)}$ is stored. The up-projection matrices $W_{UV}$ and $W_{UK}$ are absorbed directly into Query projection matrices during inference, completely eliminating the need to materialize multi-head Value caches in memory!

3.2 Decoupled RoPE (Rotary Position Embedding)

Because positional embeddings contain relative rotation matrices that cannot be absorbed into static linear up-projections, MLA introduces a dedicated decoupled Key head $k_t^{(R)} \in \mathbb{R}^{d_R}$ carrying positional encodings:

$$ext{MLA Single-Token Cache Footprint} = (d_c + d_R) imes b_{ ext{elem}}$$

Architecture	Single-Token Layer KV Cache (fp16)	Relative Memory Footprint	Expressive Capacity
MHA	$2 imes 128 imes 128 imes 2 = 65,536 ext{ bytes}$	$100\%$ (Baseline)	Full Independent Multi-Head
GQA ($n_{kv}=8$)	$2 imes 8 imes 128 imes 2 = 4,096 ext{ bytes}$	$6.25\%$	Grouped Constrained Multi-Head
DeepSeek MLA	$(512 + 64) imes 2 = 1,152 ext{ bytes}$	$1.75\%$ (98.2% Reduction)	Full Unconstrained Multi-Head

---

4. Sparse Attention Mechanics: SnapKV and PyramidKV

In ultra-long context inference (128k–1M Tokens), sparse attention algorithms provide orthogonal memory savings via dynamic eviction and layer-aware allocation:

4.1 SnapKV: Observation Window Fingerprints

• Core Insight: Attention heads exhibit highly clustered attention fingerprints. Historical Key tokens that interact frequently within an Observation Window (approx. 32 tokens at the prompt suffix) consistently dominate decoding attention weights.
• Dynamic Pruning: Based on observation window clustering scores, SnapKV evicts 80% of inactive historical tokens, retaining only 20% critical anchors with negligible perplexity degradation ($< 0.1$).

4.2 PyramidKV: Layer-Aware Hierarchical Allocation

• Core Insight: Shallow Transformer layers perform broad token extraction and require wider KV windows; deeper layers focus on high-order semantic reasoning, displaying lower sensitivity to distant historical tokens.
• Hierarchical Budgeting: PyramidKV allocates generous KV budgets to shallow layers and linearly decreases cache limits in deeper layers, slashing overall VRAM usage by 50%–70%.

---

5. Engineering Recommendations

1. High-Throughput Services & 100k+ Context: Default to native MLA models (such as DeepSeek-V3 / DeepSeek-R1) to fundamentally eliminate KV Cache bandwidth cliffs.
2. Single-GPU Edge Inference: For standard GQA models (e.g. Qwen2.5/3, Llama-3), apply SnapKV sparse kernels or 4-bit KV Cache quantization to compress 64k context footprints from 16 GB down to $<4 ext{ GB}$.
3. Anti-Pattern Warning: Avoid naive stride truncation without attention fingerprint verification, which causes critical information dropouts in multi-document retrieval tasks.