Attention Evolution & KV Cache Compression
From Multi-Head Attention (MHA) and Grouped-Query Attention (GQA) to DeepSeek Multi-Head Latent Attention (MLA) and SnapKV/PyramidKV sparse mechanics.
From Multi-Head Attention (MHA) and Grouped-Query Attention (GQA) to DeepSeek Multi-Head Latent Attention (MLA) and SnapKV/PyramidKV sparse mechanics.