Speculative Decoding: Drafting, Verification, and Real Speed Gains
When generating a long answer, the target model normally runs again after each token is chosen to obtain the probabilities for the next one. Most of each run goes to reading all the weights from GPU memory once; the arithmetic itself is a small share. In the inference-performance note, an 8-billion-parameter model on an L40S needs 18.5 ms to read its weights once but only about 0.04 ms to compute one token. Computing a few extra positions during that one read therefore costs almost no extra time.
Speculative decoding exploits this. It uses cheaper computation to propose a sequence, then has the target verify all the candidates in one run. A verification round can commit several tokens, spreading the cost of expensive serial steps across them. Leviathan et al.'s original paper gives an algorithm that preserves the target sampling distribution.
Start with prefix dependencies in autoregressive generation and the prefill, decode, and throughput boundaries in inference performance. The question here is how to reduce target calls during decode, and when fewer calls save elapsed time.
First check whether your runtime can turn it on
One person using a model on their own machine is exactly where speculative decoding pays: only one request runs at a time, and every step waits on reading the weights. The vLLM documentation also names medium-to-low QPS, memory-bound workloads as where it reduces inter-token latency. To turn it on, check which kind of draft your runtime supports, then whether your model has a matching draft.
Some models ship with a matching draft. Google publishes MTP drafters for Gemma 4, such as gemma-4-31B-it-assistant. Its card claims up to 3x faster decoding with exactly the same quality as standard generation, but shows only Transformers usage.
Even for one user it is not always faster. LM Studio's documentation says a draft that is too large or too weak can leave speed flat or make it slower; a predictable prompt such as a math question gets drafts accepted often, while open-ended story writing gets them rejected more. The draft model and its own KV cache also take memory, which a tight machine has to budget for. The test is to run the same prompts with it on and off and compare inter-token latency and results; the table at the end lists what to record.
Known candidates make parallel verification possible
In ordinary generation, choosing the second token requires the first, and choosing the third requires both. The KV cache retains the existing prefix's state; the target still processes each new position.
Let be the committed prefix and let the draft propose . All candidate prefixes are now known, so a causal target forward pass can obtain:
Verify against , against , and so on: the offset between logits and candidate positions matters. If all three are accepted, sample a bonus token from . If the second is rejected, retain only the first and add a correction at the second position. Discard the original third candidate and its verification result because they depend on the rejected prefix. EOS or an output limit can end the round early.
The target verification is parallel; a linear draft may still generate candidates sequentially. Whether the target verifies 4 positions or generates 1, it reads the weights and historical state once; the extra positions add only computation. As long as that computation stays below the compute limit, verifying a few more positions costs almost no extra time, and that is why speculative decoding can save time.
Acceptance and rejection preserve the distribution
The rule in Section 2.3 and Appendix A.1 of the original paper uses normalized target distribution and actual proposal distribution for the same prefix over the same token identifiers. Here, includes the target's temperature, top-k, top-p, or other sampling transformations; must likewise describe the draft's actual proposal procedure.
Accept a candidate sampled from with probability:
In plain words: if the target likes this token at least as much as the draft does, that is , keep it; if the draft likes it more, keep it at a discount equal to the ratio . A proposed token has . At the first rejection, sample a replacement from the remaining probability mass:
This rule says that after a rejection, the replacement is drawn only from tokens the draft under-weighted. Only tokens with positive qualify, and the more the draft under-weighted a token, the likelier it is to be drawn. When , every candidate is accepted and the zero-denominator correction branch is never entered. Applying these rules correctly at successive prefixes preserves the target sequence distribution. Rejecting low-probability candidates and simply resampling from generally changes it.
Account for all probability mass with three tokens
Consider a constructed vocabulary A, B, C with and . The draft favours A and C, so they are kept at discounts of and ; the target favours B, so B is always kept. The acceptance probabilities are therefore . Multiplying by proposal probabilities gives accepted mass and total acceptance . The remaining has to come back through the replacement draw. Of the three tokens, only B is under-weighted by the draft (), so every replacement is B, restoring .
Take one verification round: the draft proposes A, C, B. A's threshold is 5/6 and the draw is 0.7, so A is accepted; C's threshold is 2/3 and the draw is 0.9, so C is rejected and B is taken from the residual distribution. The round commits A, B. The draft's original third token B happens to match the B just added, yet it is discarded, because it was proposed under the prefix A, C.
Identical distributions do not require word-for-word identical answers with the same seed: algorithms can consume random numbers in different orders. vLLM's lossless-guarantee documentation also retains qualifications about floating-point precision, batch size, and logprob stability. These guarantees apply to the standard rejection sampler; vLLM's FLy verification deliberately relaxes verification and is not distribution preserving. The guarantee concerns decoding from a fixed target; changes introduced by target quantization or approximate caching require a separate comparison.
A good draft must also be cheap
At a fixed prefix, acceptance is . Similar draft and target distributions make acceptance more likely. The draft's ability to answer questions on its own does not substitute for this measure.
Use a simplified cost budget to decide whether speculation is worth testing. Suppose each round proposes candidates, each position has conditional acceptance probability , acceptance events are independent and identically distributed, and there is no early stopping. Including a correction or bonus token, expected committed tokens per round are:
These are the simplifying conditions used in Section 3.1 of the original paper. When acceptance varies by position and task, measure actual tokens committed per round rather than inserting an overall average acceptance rate without checking those conditions.
The following timings are all hypothetical: ordinary target decode takes 10 ms per token, , a whole target verification round takes 12 ms, and state handling takes 2 ms. With draft cost ms per token and no overlap between these stages, a cycle costs ms. Verifying 4 candidates costs only 2 ms more than one ordinary decode step because of the step in the opening paragraph: the weights are read once and the extra positions add only computation. For a sufficiently long generation, dividing that budget by gives the long-run average time per token.
gamma = 4
baseline_ms = 10
verify_ms = 12
state_ms = 2
for alpha, draft_ms in [(0.8, 1), (0.9, 3)]:
expected = sum(alpha ** j for j in range(gamma + 1))
cycle_ms = gamma * draft_ms + verify_ms + state_ms
per_token = cycle_ms / expected
speedup = baseline_ms / per_token
print(f'alpha={alpha:.1f} tokens={expected:.4f} cycle_ms={cycle_ms} '
f'ms/token={per_token:.3f} speedup={speedup:.3f}x')
alpha=0.8 tokens=3.3616 cycle_ms=18 ms/token=5.355 speedup=1.868x
alpha=0.9 tokens=4.0951 cycle_ms=26 ms/token=6.349 speedup=1.575x
The first draft costs 1 ms per token; the second costs 3 ms. The second commits more tokens on average but runs slower: about 1.575 times baseline speed, compared with 1.868 for the first. Real verification time can also vary with candidate count, batch size, context, and tree shape. Insert measured costs before tuning draft length; acceptance, drafting speed, verification, and state handling belong in the same calculation.
Chains, trees, and self-speculation describe different choices
The September 21, 2026 DeepSeek-V4 adaptation preprint discusses linear chains and candidate trees. The original LayerSkip paper provides a concrete self-speculation example: training uses layer dropout and an early-exit loss; inference drafts with early layers, then verifies and corrects with the remaining layers. Good early exits therefore cannot be assumed for an arbitrary unadapted model.
The first two rows describe candidate shape; the third describes where the draft comes from. A candidate tree ultimately commits one continuous path, with acceptance and sampling rules that must still preserve the target distribution. The chain's single-candidate correction rule cannot simply be applied to an arbitrary tree, and selecting the highest-probability branch is not target sampling. This differs from beam search, which explores prefixes to optimize a sequence score.
KV state must follow the accepted path
The KV-cache note explains reuse and compression. Speculation additionally requires cache contents to represent the committed prefix: remove a rejected suffix in a chain; in a tree, expose only the committed context and each node's ancestors, never sibling branches. Reuse shared prefixes and isolate temporary state after branching. If a newly sampled correction or bonus token has not yet passed through the model, compute its KV state later rather than reusing a rejected candidate's state.
The DeepSeek-V4 preprint explains that when a model also compresses history along the sequence, branches produce different compressed states, so the paper isolates branches in a temporary scratch pad and refreshes the chosen path afterwards. Testing DeepSeek-V4-Flash on eight GPUs, it reports trees beating matched-budget linear speculation by at most about 18.5% in decode throughput, with gains plateauing as the budget grows.
Compare matched workload and quality
The official vLLM documentation identifies medium-to-low QPS, memory-bound workloads as a setting for reducing inter-token latency. First check model, draft, and backend compatibility using local inference runtimes. Make two comparisons: speculation off versus on, then chains versus trees at a matched verification budget when linear speculation is already available.
Fix the target checkpoint, precision, tokenizer, prompt template, sampling and stopping rules, cache policy, hardware, and runtime version. Use the same prompt set and input-length distribution, record actual output lengths, and compare at the same concurrency or the same arrival rate in separate tests. Measure low-load latency separately from saturated throughput.
Acceptance measures how well the draft matches. Cycle time divided by committed tokens connects that match to speed. Report results with the measurement boundaries in inference performance to show whether savings occur in decode, the whole request, or the service as a whole.