Transformers: Original Architecture and Modern Blocks
A Transformer block does two things to every token: attention first fetches information from other positions, then a feed-forward sublayer (FFN) processes each position on its own. The FFN applies the same weights separately at each token position, so attention is the only part that mixes positions. Each step's result is added back to the original vector as a correction (the residual connection), and each sublayer is paired with a normalization.
Stack many such blocks and put token representations and position information underneath, and you have a Transformer. The 2017 paper specified an encoder–decoder, whose architecture diagram and experiments are covered in Reading “Attention Is All You Need”; common decoder-only blocks have since changed many details.
From Text to Input Vectors
A tokenizer splits text into tokens, each with an integer ID. The model uses each ID to pick a row from the embedding table , so tokens become an matrix, or [1,n,d] with a batch axis. How IDs are produced, and why they are only lookup indices, is covered in tokenization and text representations.
Attention Computes Q/K/V Head by Head
For , learned projections produce Q/K/V. With query heads and , common shapes are:
Q: [batch, h_q, sequence, d_h]
K: [batch, h_kv, sequence, d_h]
V: [batch, h_kv, sequence, d_h]
The attention formula is
Why the scores are divided by , and how the mask blocks positions, is explained in Attention.
Multi-head attention (MHA) has ; multi-query attention (MQA) uses ; intermediate grouped-query attention (GQA) uses . The GQA paper also includes MHA and MQA as endpoint cases. MQA/GQA primarily reduce KV-cache traffic while the query-head count stays the same; cache size is worked out in Attention Variants and KV-Cache Compression.
The attention formula must be applied with matching heads, not by blindly multiplying the displayed Q and K tensors. For equal-sized contiguous groups, require divisible by and put . Zero-based query head shares KV head :
With eight query heads and two KV heads, : queries 0–3 use KV head 0 and queries 4–7 use KV head 1. Each output has shape [batch, sequence, d_h]; concatenating eight outputs restores width , and a learned output projection follows before the residual addition. Explicitly repeating K/V across their query groups illustrates the semantics, but efficient kernels can share them without materializing those copies.
The FFN Works in Place; the Residual Turns Each Step into a Correction
A plain FFN is
Modern blocks often use SwiGLU or GEGLU, with separate gate and value projections. FFNs often account for most of a block's parameters and compute.
The residual connection lets a sublayer compute only a correction while the original vector passes upward unchanged; gradients can also travel this direct path back to the bottom layers, so stacks of dozens of layers still train. Residual/norm order comes in two forms:
- original Post-Norm: ;
- common Pre-Norm: .
Pre-Norm often makes deep-network training more stable, but is not mathematically equivalent to Post-Norm. LayerNorm centers and rescales variance; RMSNorm rescales by root mean square. Record epsilon, Pre/Post placement, and whether bias is included, rather than only saying “norm.”
What Is Normalized?
In a conventional Transformer, LayerNorm operates over the features of each token independently, not over batch or sequence positions. For one token , it computes , where and . RMSNorm instead computes in its usual bias-free form. For , unit scale, zero bias and ignoring epsilon for this arithmetic example, LayerNorm gives , while RMSNorm gives . Neither uses BatchNorm-style running batch statistics; these normalization calculations stay the same at training and inference.
Position Has to Be Added Separately
Without positional information or an order-dependent mask, self-attention is permutation-equivariant: permuting input rows permutes output rows. A fixed causal mask already introduces order, so this unrestricted symmetry does not apply to it.
RoPE is common in modern decoders.
Modern Decoder-Only Models Replaced Several Original Components
The right column lists common choices, not one standard. Llama 3's RMSNorm, RoPE, GQA, and SwiGLU form one documented package; it does not define every Transformer, and its overall performance cannot be credited to any single component.
For a 4096-Token Sequence, the Score Matrix Alone Takes 256 MiB
For batch=2, , , and , :
Q/K/V: [2, 8, 128, 64]
attention scores: [2, 8, 128, 128]
concatenated output: [2, 128, 512]
For batch=1, , and eight heads, each head stores one score per pair of positions, so the scores contain
elements. At fp16, each element takes 2 bytes, so the score matrix alone occupies bytes MiB, before gradients, Softmax intermediates, Q/K/V projections, FFNs, and additional layers. FlashAttention uses tiling to avoid materializing the whole intermediate matrix and reduce peak memory, but exact full-attention arithmetic remains quadratic.
Training and Generation Are Different Systems
Causal-LM training computes positions in parallel for a known sequence; generation depends token by token on prior outputs.
Both phases read results from the same exit. The stack outputs a hidden vector at each position, not a token ID. A vocabulary projection produces logits . Softmax and cross-entropy turn these scores into a distribution over vocabulary items and a training loss against the next-token target. Generation selects or samples from that distribution; its normalized axis is vocabulary items, whereas attention normalizes over keys.
The training targets are the inputs shifted left by one; an example is in encoder, decoder, and encoder–decoder families.
At generation time, process the prompt once (prefill), use its last output to select the next token, then feed that token through the stack (decode), one token after another.
With an exact cache and unchanged causal prefix, old states cannot depend on the appended token, so their K/V remain reusable: the KV cache stores them and avoids recomputing them at every step. Its main storage is proportional to the product of layer count, context length, K/V-head count, head dimension, batch size, and bytes per element. Dropout must be disabled for deterministic cached-versus-uncached comparisons. Cache size and reuse conditions are developed in Attention Variants and KV-Cache Compression.
Prefill and decode have different bottlenecks, so measure their latency and throughput separately; see inference latency and throughput.
Three Topologies Decide What Each Position Can See
The topology decides which inputs each position can see and which output head comes last. The three topologies are compared in encoder, decoder, and encoder–decoder families.
Reproducing a model takes more than these labels: tokenizer, data, objective, context curriculum, optimizer, post-training, and tool protocol often affect final behavior more than an individual block detail, so record them alongside the structural settings in the configuration card at the end.
For expert selection, load balancing, and the difference between total and active parameters, see Mixture of Experts.
Failure Modes and Counterexamples
The architecture's limits are collected here:
- A reversed causal mask leaks future tokens.
- Padding, packing, or cross-document attention can contaminate training.
- “Uses FlashAttention” alone does not establish end-to-end usability: it lowers peak memory, while the arithmetic stays quadratic.
- RoPE scaling, GQA, quantization, and fused kernels can interact and degrade performance at particular lengths or dtypes.
In long contexts, models often use information in the middle less well than at the beginning or end; see context engineering.
Minimal Reproduction Configuration Card
Every symbol on this page has a key in a Hugging Face model repository's config.json. The table maps them using Qwen3-8B's config.json:
MoE models add num_experts and num_experts_per_tok: 512 and 10 in Qwen3-Next-80B-A3B's config.json.
These numbers are enough to size the KV cache; the formula is in Attention Variants and KV-Cache Compression. For Qwen3-8B, each token stores numbers, which is 144 KiB at two bytes each. Filling all 40,960 positions takes 5.625 GiB.
config.json records structure only. The training objective, data boundary, optimizer and seeds, and the inference-time cache policy, quantization and batch size are not in it, so record them separately when reproducing or comparing models.
In Transformer Explainer, enter a short prompt and follow its tokens through attention, the MLP, and next-token probabilities. Change temperature while keeping the prompt fixed to separate the model’s logits from the sampling decision. The demonstration uses GPT-2, so compare its decoder-only blocks with the original encoder–decoder diagram in Reading “Attention Is All You Need”.