Attention Mechanism
Understand content addressing, Q/K/V, masks, multi-head variants, efficient implementations, and interpretation limits through a numerical example.
Understand content addressing, Q/K/V, masks, multi-head variants, efficient implementations, and interpretation limits through a numerical example.
How MHA, GQA and MLA set the size of the KV cache, and the context length, cache precision and prefix caching you can tune when running a model.
A section-by-section reading of Vaswani et al. (2017) covering the parallelism and long-range-dependency problem, scaled dot-product and multi-head attention, positional encoding, the training setup, and the results in Tables 1–4.
Distinguish the 2017 encoder–decoder from common Pre-Norm, RMSNorm, RoPE, GQA, and SwiGLU blocks, including training and inference costs.