Encoder, Decoder, and Encoder–Decoder: How Information Flows

Parameter count alone says little about whether BERT, a generative language model, or a sequence-conversion model fits a task. First ask what each position can see, what the model predicts during training, and which output head it uses. Together these determine its immediate suitability for classification, retrieval, or generation.

Here encoder and decoder refer to common Transformer arrangements. See Transformer blocks for their components and tokenization for their inputs.

Start with visible positions

With four input tokens, position 2 in a standard bidirectional encoder may read positions 1 through 4. A standard causal decoder lets it read only positions 1 and 2. Attention masks impose this constraint.

Structure	Input visibility	Output visibility	Common tasks
Encoder-only	All valid input positions	Usually no autoregressive output side	Classification, extraction, specially trained retrieval
Decoder-only	A causal prefix, including the prompt	Prompt and already generated tokens	Continuation, conversation, code
Encoder–decoder	Bidirectional source encoder	Generated prefix plus cross-attention to source states	Translation, summarization, conditional generation

These are common designs, not laws restricting possible tasks. A decoder can have a classification head; an encoder can participate in iterative infilling; some models use prefix masks or hybrid designs. Read the model specification instead of inferring all behavior from a BERT or GPT name.

Open full-size image

Trace the arrows from the yellow input embeddings. BERT combines both directions within each layer; the original GPT only sees preceding positions and itself. ELMo joins independently trained directional LSTMs. This historical comparison illustrates context access, not an encoder–decoder translation model.

Devlin et al., BERT (2019), Figure 3; cropped from the paper.

Encoders read before deciding

BERT pretrains a bidirectional encoder using masked-token prediction and other objectives. Hiding selected input content lets it predict from both sides without directly seeing the answer. Original BERT also used next-sentence prediction, which is not mandatory for every encoder.

Consider assigning support tickets to “billing,” “fault,” or “other.” The encoder produces [batch,length,d]. A designated classification position or pooling operation yields [batch,d], followed by a linear head:

$$z=hW+b,\qquad W\in\mathbb{R}^{d\times 3},\qquad p=\operatorname{softmax}(z).$$

The result is [batch,3]: all class scores in one forward pass. This is an architectural illustration. An arbitrary pretrained BERT vector does not already know your ticket labels. The head and, where appropriate, backbone need task-matched training; probabilities need calibration and threshold selection.

A token-level head instead predicts entity labels at each position. A suitable similarity objective and pooling can produce retrieval representations. Sharing a backbone does not make these heads or objectives interchangeable. This distinction explains local BERT classifiers, rerankers, and decision models better than “all have a few hundred million parameters.”

Decoders continue a sequence

A causal language model predicts the next token from tokens already present. During training, a full example can enter the network, while a causal mask hides future answers. Losses at many positions can therefore be computed in parallel. At inference, future tokens do not yet exist: normally the model generates one, appends it, and repeats.

For BOS I like tea EOS, one arrangement uses input BOS I like tea and targets I like tea EOS. The position predicting “tea” can read BOS I like, not the target “tea.” Some libraries shift labels internally; do not shift twice.

A vocabulary projection turns the final hidden state into [batch,V] logits, one per candidate token. This is neither a score for an entire answer nor a probability that the answer is correct. Autoregressive generation and decoding explains the text-generation loop.

Encoder–decoder separates source and target

The original Transformer performs sequence transduction; T5 casts many NLP tasks as text-to-text transformations. Suppose the source has 5 tokens and the target prefix has 3. The encoder outputs [batch,5,d]. Decoder self-attention is causal over the 3 target positions. Cross-attention uses target queries with keys and values from the 5 source states.

Its attention-score matrix therefore ends in [3,5], not [3,3]. Each target position may inspect the entire source without seeing future target tokens. Full input visibility and prevention of answer leakage are compatible.

Source states can be reused across generation steps. A decoder-only model instead places instructions, material, and output in one sequence and conditions on its prefix. Parameter allocation, caching, and training differ. Counting modules alone cannot establish which arrangement is faster or more accurate for a task.

A small visibility check

Number four positions from 0 to 3. Write two visibility matrices: all ones for bidirectional attention, and ones only where column index is no greater than row index for causal attention. Replace token 3. Earlier encoder outputs may change; earlier outputs of a standard causal decoder should not. In an actual model, disable dropout and keep positions, masks, and other inputs identical.

This detects mask errors, not successful training. A classifier evaluated with later information unavailable in production still suffers data leakage even when its architecture is implemented correctly.

Choose from the required output: a small fixed label set suggests a classifier; scoring candidate pairs suggests a reranker; free text requires a generator; explicit source-to-target conversion may suit encoder–decoder. Then compare data, languages, lengths, training, and measured cost. Architecture narrows candidates; pretraining and post-training explains how capabilities are learned.

My reading · Opened notes are remembered in this browser.