Autoregressive Generation and Decoding

At each step, a generative language model usually produces vocabulary scores. A decoder selects a token and appends it to the context. A complete answer is the result of this loop. Separating the loop’s parts distinguishes what the model learned, what the selection rule chose, and why the service stopped.

Start with token representations and causal decoders. This page concerns autoregressive text generation; alternatives such as diffusion generation do not follow the same token-by-token loop.

Factoring a sequence probability

Given prompt $x$, the probability of output $y_1,\ldots,y_T$ is:

$$p(y\mid x)=\prod_{t=1}^{T}p(y_t\mid x,y_{<t}).$$

This is the probability chain rule. An autoregressive model learns these conditional distributions; maximum-likelihood training commonly minimizes the sum of negative log probabilities. Training can evaluate many targets using their true prefixes. At inference, prefixes contain the model’s own previous outputs, so early errors change later inputs.

A projection produces $V$ logits at each step; softmax makes a next-token distribution. It describes continuation likelihood, not factual correctness. A model can confidently emit an incorrect name. Lowering temperature does not supply missing knowledge.

Greedy choices and sequence search

Greedy decoding selects the most probable token at every step. It is inexpensive and relatively easy to interpret, but need not find the most probable sequence. In a constructed two-step example, A has probability 0.6 and B has 0.4. The best ending after A has conditional probability 0.5, versus 0.9 after B. The greedy path scores $0.6\times0.5=0.30$; the alternative scores $0.4\times0.9=0.36$.

Beam search keeps several prefixes to explore a wider set of sequences. It requires additional computation and cache space. Length matters: multiplying more probabilities below 1 often lowers a longer sequence’s joint probability, so implementations commonly adjust for length. Higher likelihood also need not mean better open-ended writing. Holtzman et al. examine repetition and degeneration under likelihood-maximizing decoding and introduce nucleus sampling.

Short labels and exact extraction may favor stability; creative tasks may benefit from multiple plausible continuations. No setting is universally best.

Temperature, top-k, and top-p

Temperature transforms logits rather than directly adding creativity:

$$p_i(T)=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\qquad T>0.$$

Values below 1 concentrate the distribution; values above 1 flatten it. Zero temperature is usually an implementation’s greedy option or limiting behavior, not a valid substitution into this formula. It is also not correctness calibration: a sharper generation distribution need not lower the actual error rate.

Top-k retains k highest-scoring candidates and renormalizes. Top-p retains the shortest probability-sorted prefix whose cumulative mass reaches p, then renormalizes. For [0.50,0.25,0.15,0.10] with top-p 0.8, the first three survive, with total mass 0.90; the third probability is not sliced to 0.05. The order of filtering and temperature can change results. Follow the actual implementation; the Transformers generation documentation describes these strategies.

This example transforms hypothetical logits without calling a model:

import math

def distribution(logits, temperature):
    scaled = [v / temperature for v in logits]
    peak = max(scaled)
    weights = [math.exp(v - peak) for v in scaled]
    total = sum(weights)
    return [round(v / total, 4) for v in weights]

for t in (0.5, 1.0, 2.0):
    print(t, distribution([2.0, 1.0, 0.0], t))

Increasing temperature lowers the first probability and raises the others, without changing their ranking. Sampling can still select a low-probability token. A fixed seed helps reproducibility, but changes in serving implementation, hardware operations, and batch scheduling may still affect complete results.

Keep the logits fixed and move the temperature slider. Watch the probabilities change while the candidate order stays the same.

One set of scores, different temperatures

These are fictional candidates with logits [2, 1, 0]. Try T = 0.2, 1, and 2.

Temperature

T = 1.0

A z = 2

66.5%

B z = 1

24.5%

C z = 0

9.0%

A stays the most likely candidate. Higher temperature spreads probability more evenly; it does not add knowledge or check whether a token is correct. No model is called.

Stopping belongs to the system

Generation may end with EOS, an output-token limit, a stop string, or cancellation. Normal completion, exhausted budget, and interruption are different outcomes. Structured tasks should inspect the termination reason rather than treating truncated JSON as success.

A stop string may occur in legitimate content. Know whether a length parameter counts generated tokens or also includes the prompt. The context window must accommodate input, generated content, and any other sequence components required by the implementation. Longer outputs increase computation and cache usage. Inference latency and throughput distinguishes prompt processing from iterative decoding.

Constrained decoding can mask tokens that violate a JSON or tool-call grammar. This improves structural validity but does not establish that a date is real, an SQL query is appropriate, or tool arguments meet business constraints. The full pipeline still needs parsing, schema validation, and deterministic checks before execution.

Compare settings on a task

Fix the model, template, and input set. Compare greedy decoding, a small set of sampling configurations, and task-appropriate search. For open-ended tasks, sample repeatedly per input and preserve seeds and stopping reasons. For classification or extraction, compare accuracy, valid-format rate, latency, and cost. One impressive response is insufficient.

Measure correctness separately from repeatability. Repeating the same error ten times improves neither truth nor task success. A correct sampled answer also needs a frequency estimate. For probabilities used to trigger actions, see calibration, thresholds, and abstention; for complete answering systems, see retrieval and generation evaluation.

My reading · Opened notes are remembered in this browser.