Skip to main content

Speculative Decoding: Drafting, Verification, and Real Speed Gains

When generating a long answer, the target model normally runs again after each token is chosen to obtain the probabilities for the next one. Most of each run goes to reading all the weights from GPU memory once; the arithmetic itself is a small share. In the inference-performance note, an 8-billion-parameter model on an L40S needs 18.5 ms to read its weights once but only about 0.04 ms to compute one token. Computing a few extra positions during that one read therefore costs almost no extra time.

Speculative decoding exploits this. It uses cheaper computation to propose a sequence, then has the target verify all the candidates in one run. A verification round can commit several tokens, spreading the cost of expensive serial steps across them. Leviathan et al.'s original paper gives an algorithm that preserves the target sampling distribution.

Start with prefix dependencies in autoregressive generation and the prefill, decode, and throughput boundaries in inference performance. The question here is how to reduce target calls during decode, and when fewer calls save elapsed time.

First check whether your runtime can turn it on​

One person using a model on their own machine is exactly where speculative decoding pays: only one request runs at a time, and every step waits on reading the weights. The vLLM documentation also names medium-to-low QPS, memory-bound workloads as where it reduces inter-token latency. To turn it on, check which kind of draft your runtime supports, then whether your model has a matching draft.

RuntimeHow to turn it onWhat draft it needs
llama.cpp's llama-server-md names the draft model; --spec-type picks draft-simple, draft-eagle3, draft-mtp, draft-dflash or one of several n-gram modes; the default is none, and each round drafts 3 tokens by default (server documentation)a draft model compatible with the target; the n-gram modes find candidates in existing text and need no extra model
LM Studioin Power User mode, load a model, then select a Draft Model (documentation)the same vocabulary as the main model; the documentation suggests a draft of at most about 1B for a 7B main model, 3B for 14B, 7B for 32B
Ollamafollows the model package: Nemotron 3.5 Bolt E4B, added in 0.40.10, includes a DFlash draft, and from 0.40.12 MLX supports DFlash2 drafts (release notes)the draft that ships in the model package
vLLMmethods include EAGLE, MTP, draft models, PARD, n-gram and suffixthe documentation says model-based methods cut latency most; n-gram and suffix give modest speedups without adding load at peak traffic

Some models ship with a matching draft. Google publishes MTP drafters for Gemma 4, such as gemma-4-31B-it-assistant. Its card claims up to 3x faster decoding with exactly the same quality as standard generation, but shows only Transformers usage.

Even for one user it is not always faster. LM Studio's documentation says a draft that is too large or too weak can leave speed flat or make it slower; a predictable prompt such as a math question gets drafts accepted often, while open-ended story writing gets them rejected more. The draft model and its own KV cache also take memory, which a tight machine has to budget for. The test is to run the same prompts with it on and off and compare inter-token latency and results; the table at the end lists what to record.

Known candidates make parallel verification possible​

In ordinary generation, choosing the second token requires the first, and choosing the third requires both. The KV cache retains the existing prefix's state; the target still processes each new position.

Let hh be the committed prefix and let the draft propose x1,x2,x3x_1,x_2,x_3. All candidate prefixes are now known, so a causal target forward pass can obtain:

p1=p(⋅∣h),p2=p(⋅∣h,x1),p3=p(⋅∣h,x1,x2),p4=p(⋅∣h,x1,x2,x3).\begin{aligned} p_1&=p(\cdot\mid h),\\ p_2&=p(\cdot\mid h,x_1),\\ p_3&=p(\cdot\mid h,x_1,x_2),\\ p_4&=p(\cdot\mid h,x_1,x_2,x_3). \end{aligned}

Verify x1x_1 against p1p_1, x2x_2 against p2p_2, and so on: the offset between logits and candidate positions matters. If all three are accepted, sample a bonus token from p4p_4. If the second is rejected, retain only the first and add a correction at the second position. Discard the original third candidate and its verification result because they depend on the rejected prefix. EOS or an output limit can end the round early.

The target verification is parallel; a linear draft may still generate candidates sequentially. Whether the target verifies 4 positions or generates 1, it reads the weights and historical state once; the extra positions add only computation. As long as that computation stays below the compute limit, verifying a few more positions costs almost no extra time, and that is why speculative decoding can save time.

Acceptance and rejection preserve the distribution​

The rule in Section 2.3 and Appendix A.1 of the original paper uses normalized target distribution pp and actual proposal distribution qq for the same prefix over the same token identifiers. Here, pp includes the target's temperature, top-k, top-p, or other sampling transformations; qq must likewise describe the draft's actual proposal procedure.

Accept a candidate xx sampled from qq with probability:

a(x)=min⁡(1,p(x)q(x)).a(x)=\min\left(1,\frac{p(x)}{q(x)}\right).

In plain words: if the target likes this token at least as much as the draft does, that is p(x)≥q(x)p(x)\ge q(x), keep it; if the draft likes it more, keep it at a discount equal to the ratio p(x)/q(x)p(x)/q(x). A proposed token has q(x)>0q(x)>0. At the first rejection, sample a replacement from the remaining probability mass:

r(x)=max⁡(0,p(x)−q(x))∑ymax⁡(0,p(y)−q(y)).r(x)=\frac{\max(0,p(x)-q(x))}{\sum_y\max(0,p(y)-q(y))}.

This rule says that after a rejection, the replacement is drawn only from tokens the draft under-weighted. Only tokens with positive p(x)−q(x)p(x)-q(x) qualify, and the more the draft under-weighted a token, the likelier it is to be drawn. When p=qp=q, every candidate is accepted and the zero-denominator correction branch is never entered. Applying these rules correctly at successive prefixes preserves the target sequence distribution. Rejecting low-probability candidates and simply resampling from pp generally changes it.

Account for all probability mass with three tokens​

Consider a constructed vocabulary A, B, C with p=(0.5,0.3,0.2)p=(0.5,0.3,0.2) and q=(0.6,0.1,0.3)q=(0.6,0.1,0.3). The draft favours A and C, so they are kept at discounts of 0.5/0.6=5/60.5/0.6=5/6 and 0.2/0.3=2/30.2/0.3=2/3; the target favours B, so B is always kept. The acceptance probabilities are therefore 5/6,1,2/35/6,1,2/3. Multiplying by proposal probabilities gives accepted mass (0.5,0.1,0.2)(0.5,0.1,0.2) and total acceptance 0.80.8. The remaining 0.20.2 has to come back through the replacement draw. Of the three tokens, only B is under-weighted by the draft (0.3>0.10.3>0.1), so every replacement is B, restoring (0.5,0.3,0.2)(0.5,0.3,0.2).

Take one verification round: the draft proposes A, C, B. A's threshold is 5/6 and the draw is 0.7, so A is accepted; C's threshold is 2/3 and the draw is 0.9, so C is rejected and B is taken from the residual distribution. The round commits A, B. The draft's original third token B happens to match the B just added, yet it is discarded, because it was proposed under the prefix A, C.

Identical distributions do not require word-for-word identical answers with the same seed: algorithms can consume random numbers in different orders. vLLM's lossless-guarantee documentation also retains qualifications about floating-point precision, batch size, and logprob stability. These guarantees apply to the standard rejection sampler; vLLM's FLy verification deliberately relaxes verification and is not distribution preserving. The guarantee concerns decoding from a fixed target; changes introduced by target quantization or approximate caching require a separate comparison.

A good draft must also be cheap​

At a fixed prefix, acceptance is ∑xmin⁡(p(x),q(x))\sum_x\min(p(x),q(x)). Similar draft and target distributions make acceptance more likely. The draft's ability to answer questions on its own does not substitute for this measure.

Use a simplified cost budget to decide whether speculation is worth testing. Suppose each round proposes γ\gamma candidates, each position has conditional acceptance probability α\alpha, acceptance events are independent and identically distributed, and there is no early stopping. Including a correction or bonus token, expected committed tokens per round are:

E[N]=1+α+α2+⋯+αγ.E[N]=1+\alpha+\alpha^2+\cdots+\alpha^\gamma.

These are the simplifying conditions used in Section 3.1 of the original paper. When acceptance varies by position and task, measure actual tokens committed per round rather than inserting an overall average acceptance rate without checking those conditions.

The following timings are all hypothetical: ordinary target decode takes 10 ms per token, γ=4\gamma=4, a whole target verification round takes 12 ms, and state handling takes 2 ms. With draft cost dd ms per token and no overlap between these stages, a cycle costs 4d+12+24d+12+2 ms. Verifying 4 candidates costs only 2 ms more than one ordinary decode step because of the step in the opening paragraph: the weights are read once and the extra positions add only computation. For a sufficiently long generation, dividing that budget by E[N]E[N] gives the long-run average time per token.

gamma = 4
baseline_ms = 10
verify_ms = 12
state_ms = 2
for alpha, draft_ms in [(0.8, 1), (0.9, 3)]:
expected = sum(alpha ** j for j in range(gamma + 1))
cycle_ms = gamma * draft_ms + verify_ms + state_ms
per_token = cycle_ms / expected
speedup = baseline_ms / per_token
print(f'alpha={alpha:.1f} tokens={expected:.4f} cycle_ms={cycle_ms} '
f'ms/token={per_token:.3f} speedup={speedup:.3f}x')
alpha=0.8 tokens=3.3616 cycle_ms=18 ms/token=5.355 speedup=1.868x
alpha=0.9 tokens=4.0951 cycle_ms=26 ms/token=6.349 speedup=1.575x

The first draft costs 1 ms per token; the second costs 3 ms. The second commits more tokens on average but runs slower: about 1.575 times baseline speed, compared with 1.868 for the first. Real verification time can also vary with candidate count, batch size, context, and tree shape. Insert measured costs before tuning draft length; acceptance, drafting speed, verification, and state handling belong in the same calculation.

Chains, trees, and self-speculation describe different choices​

The September 21, 2026 DeepSeek-V4 adaptation preprint discusses linear chains and candidate trees. The original LayerSkip paper provides a concrete self-speculation example: training uses layer dropout and an early-exit loss; inference drafts with early layers, then verifies and corrects with the remaining layers. Good early exits therefore cannot be assumed for an arbitrary unadapted model.

ChoiceOrganization of candidates or computationCost to account for
Linear chainOne continuous candidate path per roundAn early rejection wastes later candidates
Candidate treeSeveral continuations branch from shared prefixes, with a bounded verification-node countTree-aware causal attention, branch state, and path commitment
Self-speculationPart of the same model drafts; the full target computation verifiesEarly-exit capability, compute reuse, and rollback

The first two rows describe candidate shape; the third describes where the draft comes from. A candidate tree ultimately commits one continuous path, with acceptance and sampling rules that must still preserve the target distribution. The chain's single-candidate correction rule cannot simply be applied to an arbitrary tree, and selecting the highest-probability branch is not target sampling. This differs from beam search, which explores prefixes to optimize a sequence score.

KV state must follow the accepted path​

The KV-cache note explains reuse and compression. Speculation additionally requires cache contents to represent the committed prefix: remove a rejected suffix in a chain; in a tree, expose only the committed context and each node's ancestors, never sibling branches. Reuse shared prefixes and isolate temporary state after branching. If a newly sampled correction or bonus token has not yet passed through the model, compute its KV state later rather than reusing a rejected candidate's state.

The DeepSeek-V4 preprint explains that when a model also compresses history along the sequence, branches produce different compressed states, so the paper isolates branches in a temporary scratch pad and refreshes the chosen path afterwards. Testing DeepSeek-V4-Flash on eight GPUs, it reports trees beating matched-budget linear speculation by at most about 18.5% in decode throughput, with gains plateauing as the budget grows.

Compare matched workload and quality​

The official vLLM documentation identifies medium-to-low QPS, memory-bound workloads as a setting for reducing inter-token latency. First check model, draft, and backend compatibility using local inference runtimes. Make two comparisons: speculation off versus on, then chains versus trees at a matched verification budget when linear speculation is already available.

Fix the target checkpoint, precision, tokenizer, prompt template, sampling and stopping rules, cache policy, hardware, and runtime version. Use the same prompt set and input-length distribution, record actual output lengths, and compare at the same concurrency or the same arrival rate in separate tests. Measure low-load latency separately from saturated throughput.

RecordHow to interpret the gain
Committed tokens per round, candidate counts, and the acceptance-rate definitionSeparate draft acceptance from useful output; total tree nodes are not accepted-path length
Drafting, verification, state-refresh time, and peak device memoryCheck whether extra accepted tokens repay extra work; include draft weights and temporary branch state
Median and p95 TTFT, token intervals, and end-to-end latencySpeculation can emit bursts; average token speed hides gaps between bursts
Final output tokens/s, completed requests/s, errors, and timeoutsRejected candidates do not count as useful throughput; retain failed requests
Accuracy, format validity, and completion rate on the same tasksDistinguish preserved target sampling from quality changes due to precision, cache, or sampling settings

Acceptance measures how well the draft matches. Cycle time divided by committed tokens connects that match to speed. Report results with the measurement boundaries in inference performance to show whether savings occur in decode, the whole request, or the service as a whole.

Explore connectionsOpen network