Autoregressive Generation and Decoding

At each step, a generative language model usually produces vocabulary scores. A decoder selects a token and appends it to the context. A complete answer is the result of this loop. Separating the loop’s parts distinguishes what the model learned, what the selection rule chose, and why the service stopped. A reasoning model “thinks” before it answers, and that thinking is tokens from the same loop, billed as output.

Factoring a sequence probability

Given prompt $x$, the probability of output $y_1,\ldots,y_T$ is:

$$p(y\mid x)=\prod_{t=1}^{T}p(y_t\mid x,y_{<t}).$$

This is the probability chain rule. An autoregressive model learns these conditional distributions; maximum-likelihood training commonly minimizes the sum of negative log probabilities. Training can evaluate many targets using their true prefixes. At inference, prefixes contain the model’s own previous outputs, so early errors change later inputs.

At each step, a projection maps the hidden vector to $V$ logits; softmax makes a next-token distribution. It describes continuation likelihood, not factual correctness. A model can confidently emit an incorrect name. Lowering temperature does not supply missing knowledge.

Greedy choices and sequence search

Greedy decoding selects the most probable token at every step. It is inexpensive and relatively easy to interpret, but need not find the most probable sequence. In a constructed two-step example, A has probability 0.6 and B has 0.4. The best ending after A has conditional probability 0.5, versus 0.9 after B. The greedy path scores $0.6\times0.5=0.30$; the alternative scores $0.4\times0.9=0.36$.

Beam search keeps several prefixes to explore a wider set of sequences. It requires additional computation and cache space. Length matters: multiplying more probabilities below 1 often lowers a longer sequence’s joint probability, so implementations commonly adjust for length. Higher likelihood also need not mean better open-ended writing. Holtzman et al. examine repetition and degeneration under likelihood-maximizing decoding and introduce nucleus sampling.

Short labels and exact extraction may favor stability; creative tasks may benefit from multiple plausible continuations. No setting is universally best.

Temperature, top-k, and top-p

These three parameters are knobs of local and open-weight runtimes: Transformers’ GenerationConfig and the llama.cpp server (--temp, --top-k, --top-p) expose them directly. Current hosted frontier reasoning models mostly restrict or reject them and use an effort setting instead; see the next section.

Temperature transforms logits rather than directly adding creativity:

$$p_i(T)=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\qquad T>0.$$

Values below 1 concentrate the distribution; values above 1 flatten it. Zero temperature is usually an implementation’s greedy option or limiting behavior; $T=0$ cannot be substituted directly into this formula. It is also not correctness calibration: a sharper generation distribution need not lower the actual error rate.

Top-k retains k highest-scoring candidates, then renormalizes and samples. Top-p retains the shortest probability-sorted prefix whose cumulative mass reaches p, then renormalizes. For [0.50,0.25,0.15,0.10] with top-p 0.8, the first three survive, with total mass 0.90; the third probability is not sliced to 0.05. The order of filtering and temperature can change results. Follow the actual implementation; the Transformers GenerationConfig reference describes these strategies.

This example transforms hypothetical logits without calling a model:

import math

def distribution(logits, temperature):
    scaled = [v / temperature for v in logits]
    peak = max(scaled)
    weights = [math.exp(v - peak) for v in scaled]
    total = sum(weights)
    return [round(v / total, 4) for v in weights]

for t in (0.5, 1.0, 2.0):
    print(t, distribution([2.0, 1.0, 0.0], t))

Increasing temperature lowers the first probability and raises the others, without changing their ranking. Sampling can still select a low-probability token. A fixed seed helps reproducibility, but changes in serving implementation, hardware operations, and batch scheduling may still affect complete results.

Keep the logits fixed and move the temperature slider. Watch the probabilities change while the candidate order stays the same.

One set of scores, different temperatures

These are fictional candidates with logits [2, 1, 0]. Try T = 0.2, 1, and 2.

Temperature

T = 1.0

A z = 2

66.5%

B z = 1

24.5%

C z = 0

9.0%

A stays the most likely candidate. Higher temperature spreads probability more evenly; it does not add knowledge or check whether a token is correct. No model is called.

Speculative decoding reduces costly serial calls by proposing candidates, then accepting or correcting them to preserve the target sampling distribution.

Hosted reasoning models use effort instead of sampling parameters

As of 2026-10-29, the two main providers work as follows. Anthropic’s thinking documentation lists current models such as Claude Fable 5.1, Opus 5.5, Sonnet 5.5, and Haiku 5.5. On these models, non-default temperature, top_p, or top_k values return a 400 error on every request; starting with API version 2026-10-21, a request that includes any of the three, at any value, returns a 400 error on every model. OpenAI’s latest-model guide says that when reasoning effort is not none, you remove temperature, top_p, and top_logprobs, and in Chat Completions also logprobs. Both providers have you tune effort instead: Anthropic’s effort sets how much work goes into the whole response, including how deeply the model thinks; OpenAI uses reasoning.effort, called reasoning_effort in Chat Completions.

Reasoning tokens come out of the same token-by-token loop. The model first generates its thinking, then the answer you see, and both count as output. Anthropic states that thinking tokens are billed as output tokens even when the thinking text isn’t returned, and that they count toward max_tokens together with the response text. OpenAI’s reasoning guide likewise says reasoning tokens are not visible through the API but occupy context-window space and are billed as output tokens; max_output_tokens caps reasoning tokens, visible output, and non-visible formatting tokens together.

So the output limit has to leave room for both thinking and answer. Set it too low and the model can stop before writing a single visible word: OpenAI then returns status: "incomplete" with incomplete_details.reason set to max_output_tokens, and the input and reasoning tokens are still billed. To spend less, lower effort first; in Anthropic’s words, effort is soft guidance and max_tokens is the strict limit.

Stopping belongs to the system

Generation may end with EOS, an output-token limit, a stop string, or cancellation. Normal completion, exhausted budget, and interruption are different outcomes. Structured tasks should inspect the termination reason rather than treating truncated JSON as success.

Hosted APIs write the termination reason into the response. Anthropic’s Messages API reports one of the stop_reason values below; OpenAI’s Responses API returns the incomplete status from the previous section when it hits the output limit or the context window.

stop_reason	When it occurs	What to do
end_turn	The model finished naturally	Use the response
max_tokens	The response reached your max_tokens	Raise the limit or continue the response
stop_sequence	The model emitted one of your stop strings	Read the stop_sequence field to see which
tool_use	The model is calling a tool	Run the tool and send the result back
pause_turn	A server-tool loop reached its iteration limit	Send the assistant content back to continue
refusal	The model declined to respond	Read stop_details; retry on a fallback model if appropriate
model_context_window_exceeded	The response filled the context window	Treat it as truncated

A stop string may occur in legitimate content. Know whether a length parameter counts generated tokens or also includes the prompt. To limit only newly generated output, use the interface’s output-length parameter. The context window must accommodate input, generated content, and any other sequence components required by the implementation. Longer outputs increase computation and cache usage. Inference latency and throughput distinguishes prompt processing from iterative decoding.

Constrained decoding can mask tokens that violate a JSON or tool-call grammar. This improves structural validity but does not establish that a date is real, an SQL query is appropriate, or tool arguments meet business constraints. The full pipeline still needs parsing, schema validation, and deterministic checks before execution.

Compare settings on a task

Fix the model, template, and input set, then compare a few settings; on hosted reasoning models, compare effort levels. For open-ended tasks, generate several times per input and record stop reasons; for classification or extraction, compare accuracy, valid-format rate, latency, and cost. A setting that repeats the same error ten times is more stable, not more correct. For probabilities used to trigger actions, see calibration, thresholds, and abstention; for complete answering systems, see retrieval and generation evaluation.

My reading