Inference Performance: Loading, First Tokens, Decoding, and Throughput
Tokens per second describe only part of inference. A fast generator can still make users wait for long-prompt processing, queues, or model loading. Before comparing runtimes, decide whether the workload is one local user or many concurrent requests, and whether first response or whole-batch completion matters most.
Separate request stages
A request passes through several waits between being sent and finishing. If the model is not yet loaded, the runtime first reads files from disk, allocates memory and moves the weights to the GPU; some runtimes also compile or prepare kernels. This is the cold start. Incoming text is then split into tokens (tokenization), and server queues, networks and client buffering add to the delay the user feels.
The model's own work has two phases. Prefill reads the whole prompt at once and builds the state that generation needs. Decode, in ordinary autoregressive generation, appends output tokens one at a time. In the timeline in the next section, the first token appears 0.8 seconds after the request; those 0.8 seconds go mostly to prefill. After that, one token every 0.02 seconds is the decode rhythm.
The vLLM serving benchmark exposes first-token, per-output-token, inter-token, and end-to-end metrics with percentiles. Other tools need equally explicit boundaries, including whether client queuing is counted.
Speculative decoding can commit multiple tokens per verification round; include drafting, verification, and state refresh in the timing.
Work through a timeline
Suppose a request starts at 0 seconds, its first token arrives at 0.8 seconds, and its 101st and final token arrives at 2.8 seconds. Ignoring finalization, TTFT is 0.8 seconds. Mean time per subsequent token is (2.8 - 0.8) / (101 - 1) = 0.02 seconds, equivalent to 50 tokens/s during that decode interval. Dividing all output tokens by total request time instead gives 101 / 2.8, about 36.1 tokens/s. Both are defined quantities; they are not interchangeable.
Real streaming chunks may contain multiple tokens, so counting chunks can overestimate or underestimate speed. Count with the model's tokenizer or reliable server usage rather than counting chunks, and retain the measurement definition.
Every generated token reads all the weights once
Where decode time goes explains why batching, quantization, speculative decoding and the KV cache matter. The answer: for every token it generates, the model reads all of its weights from GPU memory once, and reading takes far longer than computing.
First, why all the weights. To produce the next token, the vector of the token just generated passes through every layer, and each layer multiplies it by its own weight matrices. Every weight is used once; when that token is done, the next one has to read every weight again.
Second, which is slower, reading or computing. Take an 8-billion-parameter model such as Llama 3 8B, stored in BF16 at 2 bytes per parameter: 8×10⁹ × 2 = 16 GB of weights. On an NVIDIA L40S, whose listed specifications give 864 GB/s of memory bandwidth and 362 trillion BF16 tensor operations per second:
- Reading the weights once: 16 / 864 ≈ 0.0185 s, or 18.5 ms.
- Computing one token: one multiply and one add per parameter, 16×10⁹ operations; at 362×10¹² per second that takes about 0.04 ms.
Reading is over four hundred times slower, so when one person generates token by token, speed depends almost entirely on how large the weights are and how fast memory can be read. Even if computation took no time, this model on this card could produce at most 1 / 0.0185 ≈ 54 tokens per second. The timeline above, one token every 0.02 seconds or 50 per second, sits right at that scale.
On your own machine it is the same division. Apple lists the M5 Pro's memory bandwidth at 307 GB per second. Llama 3.1 8B quantized to Q4_K_M is about 4.9 GB (see the quantization note), so the ceiling is about 307 ÷ 4.9 ≈ 63 tokens/s.
Mixture-of-experts (MoE) models are the exception. Each token passes only through the few experts the router selects, so the bytes read per step follow the active parameters, far fewer than all the weights. The unselected experts still have to sit in memory, though, so capacity still follows the total parameters; see mixture of experts.
The common ways to speed up decoding or save memory all work on this one step:
Prefill escapes this limit. The thousands of tokens in a prompt are computed together, so one read of the weights serves all of them, and compute becomes the bottleneck. Long prompts therefore mainly delay the first token, while long outputs mainly stretch decode.
Batching changes the tradeoff
Batching uses exactly that step: one read of the weights can compute one token for each of several users. Keep the same model and card, and give each user a 4K-token context. With Llama 3 8B's shape, each token's KV cache takes 128 KiB, so 4K tokens take 512 MiB, about 0.54 GB (worked out in the KV-cache note):
Decoding eight users together raises aggregate throughput more than sixfold, yet each user's token interval stretches from 19.1 ms to 23.5 ms. The weights are read once, but every user brings a cache of their own, so a larger batch reads more cache per step and pushes the arithmetic toward the compute limit, and each individual user slows down.
Beyond slower steps, a single request may also wait for a batch to form. Continuous batching lets completed sequences leave and new ones join without waiting for the longest sequence.
Concurrency also increases KV-cache demand. PagedAttention uses paged management for dynamically changing cache allocations, reducing fragmentation and duplication. This addresses a memory-management problem; it does not remove weight, cache-capacity, or compute limits.
Long-input, short-output tasks emphasize prefill and TTFT. Short-input, long-output tasks expose decode costs. Lower precision, larger batches, and shorter contexts change different costs and cannot substitute for one another: lower precision reads fewer weight bytes, a larger batch shares one weight read among more users, and a shorter context reads less cache per step.
Report cache hits separately
Model residency, warmed kernels, and prefix caching are different kinds of warmth. A loaded model does not imply a cached prompt; repeated identical system text may make subsequent requests faster. Compare cold starts, resident execution without prefix hits, and realistic prefix-hit workloads separately.
Reuse requires matching inputs and model configuration. Sending one identical prompt repeatedly measures ideal repetition, not changing tasks. Model, adapter, or context changes must not leave obsolete cache entries treated as valid.
Retain failures and tail latency
The estimates above from listed bandwidth and compute are ideal lower bounds; measure the real bottleneck on the target hardware. Fix the model, quantization, runtime, input lengths and concurrency, record median and high-percentile latency, and keep timeouts and failures in the results; arrival rate and load generation for a multi-user service are covered in local model evaluation.
Finally, measure whether answers are correct, tools complete their tasks, and retries are necessary. High token throughput cannot compensate for time spent on incorrect work. See runtime selection for compatibility and local model and agent evaluation for task outcomes.