Inference Performance: Loading, First Tokens, Decoding, and Throughput
Tokens per second describe only part of inference. A fast generator can still make users wait for long-prompt processing, queues, or model loading. Before comparing runtimes, decide whether the workload is one local user or many concurrent requests, and whether first response or whole-batch completion matters most.
Separate request stages
A nonresident model needs file reads, memory allocation, device transfer, and possibly compilation or kernel preparation. Text also needs tokenization. Prefill processes existing input tokens and builds the state used for generation; ordinary autoregressive decode appends output tokens sequentially. Server queues, networks, and client buffering add user-visible delay.
The vLLM serving benchmark exposes first-token, per-output-token, inter-token, and end-to-end metrics with percentiles. Other tools need equally explicit boundaries, including whether client queuing is counted.
Work through a timeline
Suppose a request starts at 0 seconds, its first token arrives at 0.8 seconds, and its 101st and final token arrives at 2.8 seconds. Ignoring finalization, TTFT is 0.8 seconds. Mean time per subsequent token is (2.8 - 0.8) / (101 - 1) = 0.02 seconds, equivalent to 50 tokens/s during that decode interval. Dividing all output tokens by total request time instead gives about 36.1 tokens/s. Both are defined quantities; they are not interchangeable.
This invented timeline is not a hardware result. Real streaming chunks may contain multiple tokens. Count with the model's tokenizer or reliable server usage rather than counting chunks, and retain the measurement definition.
Batching changes the tradeoff
Combining requests can improve utilization of weight reads and compute, but may add time waiting for a batch. Continuous batching lets completed sequences leave and new ones join without waiting for the longest sequence. Aggregate throughput can rise while each user's token interval grows.
Concurrency also increases KV-cache demand. PagedAttention uses paged management for dynamically changing cache allocations, reducing fragmentation and duplication. This addresses a memory-management problem; it does not remove weight, cache-capacity, or compute limits.
Long-input, short-output tasks emphasize prefill and TTFT. Short-input, long-output tasks expose decode costs. Small-batch decode often faces weight-traffic limits, while larger prefill can use matrix compute more fully; actual bottlenecks depend on hardware and kernels. Lower precision, larger batches, and shorter contexts change different costs.
Open full-size imageFollow logical blocks 0, 1 and 2 through the table to physical blocks 7, 1 and 3. Tokens stay in sequence order while their cache occupies scattered blocks. When the last block fills, another can be allocated. The four-token block size is illustrative; paging manages allocation rather than compressing each token’s KV vectors.
Report cache hits separately
Model residency, warmed kernels, and prefix caching are different kinds of warmth. A loaded model does not imply a cached prompt; repeated identical system text may make subsequent requests faster. Compare cold starts, resident execution without prefix hits, and realistic prefix-hit workloads separately.
Reuse requires matching inputs and model configuration. Sending one identical prompt repeatedly measures ideal repetition, not changing tasks. Model, adapter, or context changes must not leave obsolete cache entries treated as valid.
Retain failures and tail latency
Fix model version, quantization, hardware, runtime, input-length distribution, output limits, and concurrency. Mix short and long target requests, separate warmup from measurement, and report median and high-percentile latency alongside timeouts, failures, actual output lengths, and peak memory. Keeping only successful short outputs makes a system look artificially fast.
As offered load rises, queues can grow sharply. A client that sends a new request only after the previous one finishes automatically slows its offered load and can hide overload. Choose fixed concurrency or a fixed arrival rate to match the intended scenario, and state which. A p99 from very few requests is unstable and should be read with the sample size.
Finally, measure whether answers are correct, tools complete their tasks, and retries are necessary. High token throughput cannot compensate for time spent on incorrect work. See runtime selection for compatibility and local model and agent evaluation for task outcomes.