Skip to main content

Transformers: Original Architecture and Modern Blocks

A Transformer block does two things to every token: attention first fetches information from other positions, then a feed-forward sublayer (FFN) processes each position on its own. The FFN applies the same weights separately at each token position, so attention is the only part that mixes positions. Each step's result is added back to the original vector as a correction (the residual connection), and each sublayer is paired with a normalization.

Stack many such blocks and put token representations and position information underneath, and you have a Transformer. The 2017 paper specified an encoder–decoder, whose architecture diagram and experiments are covered in Reading “Attention Is All You Need”; common decoder-only blocks have since changed many details.

From Text to Input Vectors​

A tokenizer splits text into tokens, each with an integer ID. The model uses each ID to pick a row from the embedding table E∈R∣V∣×dE\in\mathbb R^{|\mathcal V|\times d}, so nn tokens become an n×dn\times d matrix, or [1,n,d] with a batch axis. How IDs are produced, and why they are only lookup indices, is covered in tokenization and text representations.

Attention Computes Q/K/V Head by Head​

For X∈Rb×n×dX\in\mathbb{R}^{b\times n\times d}, learned projections produce Q/K/V. With hqh_q query heads and dh=d/hqd_h=d/h_q, common shapes are:

Q: [batch, h_q, sequence, d_h]
K: [batch, h_kv, sequence, d_h]
V: [batch, h_kv, sequence, d_h]

The attention formula is

Attention⁡(Q,K,V)=softmax⁡ ⁣(QK⊤dh+M)V.\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_h}}+M\right)V.

Why the scores are divided by dh\sqrt{d_h}, and how the mask MM blocks positions, is explained in Attention.

Multi-head attention (MHA) has hq=hkvh_q=h_{kv}; multi-query attention (MQA) uses hkv=1h_{kv}=1; intermediate grouped-query attention (GQA) uses 1<hkv<hq1<h_{kv}<h_q. The GQA paper also includes MHA and MQA as endpoint cases. MQA/GQA primarily reduce KV-cache traffic while the query-head count stays the same; cache size is worked out in Attention Variants and KV-Cache Compression.

The attention formula must be applied with matching heads, not by blindly multiplying the displayed Q and K tensors. For equal-sized contiguous groups, require hqh_q divisible by hkvh_{kv} and put r=hq/hkvr=h_q/h_{kv}. Zero-based query head ii shares KV head g(i)=⌊i/r⌋g(i)=\lfloor i/r\rfloor:

Oi=softmax⁡ ⁣(QiKg(i)Tdh+M)Vg(i).O_i=\operatorname{softmax}\!\left(\frac{Q_iK_{g(i)}^T}{\sqrt{d_h}}+M\right)V_{g(i)}.

With eight query heads and two KV heads, r=8/2=4r=8/2=4: queries 0–3 use KV head 0 and queries 4–7 use KV head 1. Each output has shape [batch, sequence, d_h]; concatenating eight outputs restores width dd, and a learned output projection follows before the residual addition. Explicitly repeating K/V across their query groups illustrates the semantics, but efficient kernels can share them without materializing those copies.

The FFN Works in Place; the Residual Turns Each Step into a Correction​

A plain FFN is

FFN⁡(x)=W2ϕ(W1x+b1)+b2.\operatorname{FFN}(x)=W_2\phi(W_1x+b_1)+b_2.

Modern blocks often use SwiGLU or GEGLU, with separate gate and value projections. FFNs often account for most of a block's parameters and compute.

The residual connection lets a sublayer compute only a correction while the original vector passes upward unchanged; gradients can also travel this direct path back to the bottom layers, so stacks of dozens of layers still train. Residual/norm order comes in two forms:

  • original Post-Norm: y=LN⁡(x+Sublayer⁡(x))y=\operatorname{LN}(x+\operatorname{Sublayer}(x));
  • common Pre-Norm: y=x+Sublayer⁡(Norm⁡(x))y=x+\operatorname{Sublayer}(\operatorname{Norm}(x)).

Pre-Norm often makes deep-network training more stable, but is not mathematically equivalent to Post-Norm. LayerNorm centers and rescales variance; RMSNorm rescales by root mean square. Record epsilon, Pre/Post placement, and whether bias is included, rather than only saying “norm.”

What Is Normalized?​

In a conventional Transformer, LayerNorm operates over the dd features of each token independently, not over batch or sequence positions. For one token xx, it computes γ⊙(x−μ)/v+ϵ+β\gamma\odot(x-\mu)/\sqrt{v+\epsilon}+\beta, where μ=d−1∑ixi\mu=d^{-1}\sum_i x_i and v=d−1∑i(xi−μ)2v=d^{-1}\sum_i(x_i-\mu)^2. RMSNorm instead computes γ⊙x/d−1∑ixi2+ϵ\gamma\odot x/\sqrt{d^{-1}\sum_i x_i^2+\epsilon} in its usual bias-free form. For x=[1,3]x=[1,3], unit scale, zero bias and ignoring epsilon for this arithmetic example, LayerNorm gives [−1,1][-1,1], while RMSNorm gives [1,3]/5[1,3]/\sqrt5. Neither uses BatchNorm-style running batch statistics; these normalization calculations stay the same at training and inference.

Position Has to Be Added Separately​

Without positional information or an order-dependent mask, self-attention is permutation-equivariant: permuting input rows permutes output rows. A fixed causal mask already introduces order, so this unrestricted symmetry does not apply to it.

MethodWhere it actsBoundary
absolute position vectoradded to token representationextrapolation and length limits depend on the implementation
relative biasmodifies attention scoretied to bias form/window
RoPErotates pairs of coordinates in Q and K so dot products carry relative positionbase, scaling, and extrapolation must be stated
ALiBi-like biasdistance-dependent score biasa particular inductive bias, not universally best

RoPE is common in modern decoders.

Modern Decoder-Only Models Replaced Several Original Components​

ComponentOriginal Transformer (2017)Some modern decoder-only instances
topologyencoder–decodercausal decoder stack
normPost-LayerNormPre-RMSNorm or Pre-LayerNorm
positionsinusoidal absoluteRoPE or relative bias
FFNReLUGELU, SwiGLU/GEGLU, sometimes MoE
headsMHAMHA, MQA, or GQA
kernelconventional matricesfused SDPA, FlashAttention

The right column lists common choices, not one standard. Llama 3's RMSNorm, RoPE, GQA, and SwiGLU form one documented package; it does not define every Transformer, and its overall performance cannot be credited to any single component.

For a 4096-Token Sequence, the Score Matrix Alone Takes 256 MiB​

For batch=2, n=128n=128, d=512d=512, and h=8h=8, dh=512/8=64d_h=512/8=64:

Q/K/V: [2, 8, 128, 64]
attention scores: [2, 8, 128, 128]
concatenated output: [2, 128, 512]

For batch=1, n=4096n=4096, and eight heads, each head stores one score per pair of positions, so the scores contain

8×40962=134,217,7288\times4096^2=134{,}217{,}728

elements. At fp16, each element takes 2 bytes, so the score matrix alone occupies 134,217,728×2134{,}217{,}728\times2 bytes =256=256 MiB, before gradients, Softmax intermediates, Q/K/V projections, FFNs, and additional layers. FlashAttention uses tiling to avoid materializing the whole intermediate matrix and reduce peak memory, but exact full-attention arithmetic remains quadratic.

Training and Generation Are Different Systems​

Causal-LM training computes positions in parallel for a known sequence; generation depends token by token on prior outputs.

Both phases read results from the same exit. The stack outputs a hidden vector ht∈Rdh_t\in\mathbb R^d at each position, not a token ID. A vocabulary projection Wvocab∈Rd×∣V∣W_{\mathrm{vocab}}\in\mathbb R^{d\times|\mathcal V|} produces logits zt=WvocabTht+bz_t=W_{\mathrm{vocab}}^Th_t+b. Softmax and cross-entropy turn these scores into a distribution over vocabulary items and a training loss against the next-token target. Generation selects or samples from that distribution; its normalized axis is vocabulary items, whereas attention normalizes over keys.

The training targets are the inputs shifted left by one; an example is in encoder, decoder, and encoder–decoder families.

At generation time, process the prompt once (prefill), use its last output to select the next token, then feed that token through the stack (decode), one token after another.

With an exact cache and unchanged causal prefix, old states cannot depend on the appended token, so their K/V remain reusable: the KV cache stores them and avoids recomputing them at every step. Its main storage is proportional to the product of layer count, context length, K/V-head count, head dimension, batch size, and bytes per element. Dropout must be disabled for deterministic cached-versus-uncached comparisons. Cache size and reuse conditions are developed in Attention Variants and KV-Cache Compression.

Prefill and decode have different bottlenecks, so measure their latency and throughput separately; see inference latency and throughput.

Three Topologies Decide What Each Position Can See​

The topology decides which inputs each position can see and which output head comes last. The three topologies are compared in encoder, decoder, and encoder–decoder families.

Reproducing a model takes more than these labels: tokenizer, data, objective, context curriculum, optimizer, post-training, and tool protocol often affect final behavior more than an individual block detail, so record them alongside the structural settings in the configuration card at the end.

For expert selection, load balancing, and the difference between total and active parameters, see Mixture of Experts.

Failure Modes and Counterexamples​

The architecture's limits are collected here:

  • A reversed causal mask leaks future tokens.
  • Padding, packing, or cross-document attention can contaminate training.
  • “Uses FlashAttention” alone does not establish end-to-end usability: it lowers peak memory, while the arithmetic stays quadratic.
  • RoPE scaling, GQA, quantization, and fused kernels can interact and degrade performance at particular lengths or dtypes.

In long contexts, models often use information in the middle less well than at the beginning or end; see context engineering.

Minimal Reproduction Configuration Card​

Every symbol on this page has a key in a Hugging Face model repository's config.json. The table maps them using Qwen3-8B's config.json:

Symbol on this pageconfig.json keyQwen3-8B value
topologyarchitecturesQwen3ForCausalLM (causal decoder)
layers LLnum_hidden_layers36
width ddhidden_size4096
query heads hqh_qnum_attention_heads32
KV heads hkvh_{kv}num_key_value_heads8, so r=32/8=4r=32/8=4
head dimension dhd_hhead_dim128
FFN width and activationintermediate_size, hidden_act12288, silu
normalizationrms_norm_eps1e-06 (RMSNorm)
position encodingrope_theta, rope_scaling, max_position_embeddings1000000, null, 40960
vocabulary and output projectionvocab_size, tie_word_embeddings151936, false
weight precisiontorch_dtypebfloat16

MoE models add num_experts and num_experts_per_tok: 512 and 10 in Qwen3-Next-80B-A3B's config.json.

These numbers are enough to size the KV cache; the formula is in Attention Variants and KV-Cache Compression. For Qwen3-8B, each token stores 2×36×8×128=73,7282\times36\times8\times128=73{,}728 numbers, which is 144 KiB at two bytes each. Filling all 40,960 positions takes 5.625 GiB.

config.json records structure only. The training objective, data boundary, optimizer and seeds, and the inference-time cache policy, quantization and batch size are not in it, so record them separately when reproducing or comparing models.

In Transformer Explainer, enter a short prompt and follow its tokens through attention, the MLP, and next-token probabilities. Change temperature while keeping the prompt fixed to separate the model’s logits from the sampling decision. The demonstration uses GPT-2, so compare its decoder-only blocks with the original encoder–decoder diagram in Reading “Attention Is All You Need”.

Explore connectionsOpen network