Attention Mechanism

Attention is differentiable content addressing: a query scores keys, normalized weights mix the corresponding values.

$$\operatorname{Attention}(\mathbf{q},K,V)=\sum_i\alpha_i\mathbf{v}_i, \qquad \alpha_i=\frac{\exp a(\mathbf{q},\mathbf{k}_i)}{\sum_j\exp a(\mathbf{q},\mathbf{k}_j)}.$$

Query, key, and value are computational roles, not intrinsic meanings. They are usually learned projections, and values need not equal keys.

Scores

Additive attention uses a small learned network:

$$a(\mathbf{q},\mathbf{k})=\mathbf{v}^{\top}\tanh(W_q\mathbf{q}+W_k\mathbf{k}).$$

Scaled dot-product attention uses

$$a(\mathbf{q},\mathbf{k})=\frac{\mathbf{q}^{\top}\mathbf{k}}{\sqrt{d_k}}.$$

Under simple unit-variance assumptions, the unscaled dot product grows with $d_k$ and can saturate Softmax. The scaling is a design for this score, not a law for all attention.

Worked Example: A Mixture, Not a Hard Lookup

Let $\mathbf{q}=[1,0]$, $\mathbf{k}_1=[1,0]$, $\mathbf{k}_2=[0,1]$, $\mathbf{v}_1=[2,0]$, and $\mathbf{v}_2=[0,4]$. With $1/\sqrt{2}$ scaling, scores are approximately $[0.707,0]$, weights $[0.67,0.33]$, and output

$$0.67[2,0]+0.33[0,4]\approx[1.34,1.32].$$

The best key does not erase the other value. Temperature, duplicate keys, score gaps, and masks all change the mixture.

Matrix Form and Masks

For $Q\in\mathbb{R}^{n_q\times d_k}$, $K\in\mathbb{R}^{n_k\times d_k}$, and $V\in\mathbb{R}^{n_k\times d_v}$:

$$A=\operatorname{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}+M\right), \qquad O=AV.$$

$A$ has shape $n_q\times n_k$ and $O$ has shape $n_q\times d_v$. Padding, causal, and structural masks act before Softmax. An entirely masked row can produce undefined values, so safe behavior must be specified. Changing a masked token should not change an unmasked output.

Self-attention draws Q/K/V from one representation; cross-attention draws queries from one side and K/V from another. Multi-head attention learns multiple projected channels, but no head is guaranteed to map to a stable human concept.

Bahdanau-style attention first gave each RNN decoder step access to multiple encoder states; Luong compared global and local forms. The Transformer later made self-attention a primary layer operation. Attention did not begin with the Transformer.

Efficiency Designs Solve Different Problems

Design	Change	Does not automatically solve
FlashAttention	tiled, IO-aware exact attention without storing the full intermediate matrix	arithmetic remains usually quadratic; kernel/hardware dependence
MQA	query heads share one K/V head	capacity and quality trade-off
GQA	query heads share a smaller set of K/V heads	still only a cache/quality compromise
sliding-window/sparse	restrict reachable positions	omitted positions need cross-layer paths
linear/kernel attention	reorder or approximate aggregation	normalization and quality need not equal Softmax

MQA/GQA mainly reduce autoregressive KV-cache traffic. Report query heads, K/V heads, head dimension, window, dtype, and kernel rather than only saying “efficient attention.” RNNs/SSMs may suit fixed-state streaming; convolution may suit strong locality.

Interpretation Boundary

Attention weights are routing coefficients in one forward pass, but are not automatically causal importance. Different weights may yield similar outputs; values and later layers change influence; gradients and interventions may disagree; and sharpness alone says little.

“Attention is not Explanation” demonstrates failures of direct interpretation; a competing analysis argues attention can still be informative under defined diagnostics and counterfactual tests. The defensible conclusion is that weights may be evidence, but should be paired with deletion, replacement, or intervention—not treated as a standalone explanation.

Minimal Acceptance

Write every Q/K/V and score shape; test masks and all-masked rows; test permutation or duplicate-key invariants; report sequence length, heads, dtype, memory, and implementation; use interventions for token-dependence claims; and compare against pooling, recurrence, or local convolution.

This Q/K/V account is biased toward NLP and Transformer practice. Vision, sets, graphs, and earlier statistical kernel formulations receive only boundary coverage.