Transformers: Original Architecture and Modern Blocks

A Transformer is not synonymous with attention. It combines token representations, position, attention, feed-forward sublayers, residual paths, and normalization. The 2017 paper specified an encoder–decoder; common decoder-only blocks have since changed many details.

Core Tensors

For $X\in\mathbb{R}^{b\times n\times d}$, learned projections produce Q/K/V. With $h$ query heads and $d_h=d/h$, common shapes are:

Q: [batch, h_q, sequence, d_h]
K: [batch, h_kv, sequence, d_h]
V: [batch, h_kv, sequence, d_h]

$$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_h}}+M\right)V.$$

MHA has $h_q=h_{kv}$; MQA often uses $h_{kv}=1$; GQA uses $1<h_{kv}<h_q$. MQA/GQA primarily reduce KV-cache traffic, not query count, and do not guarantee unchanged quality.

A Block Is More Than Attention

A plain FFN is

$$\operatorname{FFN}(x)=W_2\phi(W_1x+b_1)+b_2.$$

Modern blocks often use SwiGLU or GEGLU, with separate gate and value projections. FFNs often contain much of a block's parameters and compute.

Residual/norm order also varies:

• original Post-Norm: $y=\operatorname{LN}(x+\operatorname{Sublayer}(x))$;
• common Pre-Norm: $y=x+\operatorname{Sublayer}(\operatorname{Norm}(x))$.

Pre-Norm often eases deep optimization but is not equivalent to Post-Norm. LayerNorm centers and rescales variance; RMSNorm rescales by root mean square. Record epsilon, placement, and bias rather than only “norm.”

Position

Self-attention alone is permutation-equivariant.

Method	Where it acts	Boundary
absolute position vector	added to token representation	extrapolation and limits vary
relative bias	modifies attention score	tied to bias form/window
RoPE	rotates Q/K pairs so dot products carry relative position	base, scaling, and extrapolation must be stated
ALiBi-like bias	distance-dependent score bias	a particular inductive bias, not universally best

RoPE is common in modern decoders. A configured long window does not prove effective retrieval, and position scaling may trade short-context quality.

Original Versus Common Modern Instances

Component	Original Transformer	Some modern decoder-only instances
topology	encoder–decoder	causal decoder stack
norm	Post-LayerNorm	Pre-RMSNorm or Pre-LayerNorm
position	sinusoidal absolute	RoPE or relative bias
FFN	ReLU	GELU, SwiGLU/GEGLU, sometimes MoE
heads	MHA	MHA, MQA, or GQA
kernel	conventional matrices	fused SDPA, FlashAttention

The right column is not one standard. Llama 3's RMSNorm, RoPE, GQA, and SwiGLU form one documented package; it neither defines every Transformer nor isolates one component's contribution.

Worked Shape and Memory Estimate

For batch=2, $n=128$, $d=512$, and $h=8$, $d_h=64$:

Q/K/V: [2, 8, 128, 64]
attention scores: [2, 8, 128, 128]
concatenated output: [2, 128, 512]

For batch=1, $n=4096$, and eight heads, scores contain

$$8\times4096^2=134{,}217{,}728$$

elements—about 256 MiB at fp16 before gradients, Softmax, Q/K/V, FFNs, and layers. FlashAttention avoids materializing this whole intermediate through tiling, but exact full-attention arithmetic remains quadratic.

Training and Generation Are Different Systems

Causal-LM training computes positions in parallel for a known sequence; generation depends token by token on prior outputs. KV caching avoids recomputing old K/V but grows with layers, context, K/V heads, head dimension, batch, and dtype.

Report prefill and decode latency separately; batch, prompt/output lengths, sampling, cache quantization/window/paging/offload, training activation versus inference resident memory, kernels, compiler, hardware, and precision. “1M context” or “uses FlashAttention” does not establish end-to-end usability.

Architecture Families

Encoder-only models use bidirectional context; decoder-only models use causal objectives; encoder–decoders separate input encoding and conditional generation. MoE Transformers route tokens through subsets of experts, increasing total capacity while introducing load balancing, communication, and routing failures.

These labels remain insufficient for reproduction: tokenizer, data, objective, context curriculum, optimizer, post-training, and tool protocol can dominate a block detail.

Failure Modes and Counterexamples

A reversed causal mask leaks future tokens. Padding, packing, or cross-document attention can contaminate training. Long contexts may underuse middle positions. Benchmark or corpus contamination is not repaired by architecture. RoPE scaling, GQA, quantization, and fused kernels can interact at specific lengths or dtypes. Fixed-memory streaming may favor RNNs/SSMs; local vision may favor convolution.

architecture: encoder | decoder | encoder-decoder
layers: N
width: d_model
attention: heads, kv_heads, head_dim, window, kernel
position: method, base, scaling, trained_length
norm: type, pre_or_post, epsilon
ffn: activation_or_gate, intermediate_width, experts
training: objective, data boundary, precision, optimizer, seeds
inference: cache, quantization, batch, prompt/output lengths

Current products and APIs belong in the Frontier Radar; this page maintains reusable architectural boundaries.