Model Quantization: Precision, Size, and Speed
Quantization approximates numerical values with fewer bits. Its most direct benefit is less data to store and move. Speed and task-quality changes depend on the model, method, kernels, and hardware. Start with a memory budget, then identify what to compress rather than relying on “4-bit” in a filename.
Floating-Point Numbers and Stable Computation explains how representation precision, numerical range, and arithmetic error differ.
Approximate a group of values
In symmetric uniform quantization, choose a scale s, encode a weight as q = round(w / s) within a finite integer range, and reconstruct it as s × q. Out-of-range values are clipped. Asymmetric quantization adds a zero point so the integer interval can represent a range not centered on zero.
This invented example uses a simplified symmetric 4-bit convention from -7 to 7, leaving one code unused. Real formats can use other layouts.
weights = [-1.0, -0.2, 0.3, 1.0]
scale = max(abs(w) for w in weights) / 7
quantized = [max(-7, min(7, round(w / scale))) for w in weights]
restored = [round(q * scale, 3) for q in quantized]
print(quantized) # [-7, -1, 2, 7]
print(restored) # [-1.0, -0.143, 0.286, 1.0]
The middle weights change: -0.2 becomes -0.143, an error of 0.057; 0.3 becomes 0.286, an error of 0.014.
With one scale for a whole layer, a large outlier makes ordinary small weights coarse. Keep the same numbers but replace 1.0 with 3.5. The scale grows from 1 / 7 ≈ 0.143 to 3.5 / 7 = 0.5. Then -0.2 / 0.5 = -0.4 rounds to 0, and 0.3 / 0.5 = 0.6 rounds to 1, restored as 0.5. Both ordinary weights now carry an error of 0.2, more than three times and fourteen times their earlier errors. That is why per-channel or smaller-group scales better match local ranges but add metadata and affect kernel costs.
Drag a weight, or select it and press ↑ ↓ (← → picks the neighbour). Each group has one scale, and every weight snaps to the nearest level of its group's grid.
Weight 7, group 2q = round(w / s) = round(0.08 / 0.101) = 1 → ŵ = s × q = 0.101 (error +0.021)
| Group | min … max | scale s |
|---|---|---|
| 1 | −1.00 … 1.00 | max|w| / 7 = 1.00 / 7 = 0.143 |
| 2 | −0.55 … 0.71 | max|w| / 7 = 0.71 / 7 = 0.101 |
| 3 | −0.86 … 0.50 | max|w| / 7 = 0.86 / 7 = 0.123 |
| 4 | −0.47 … 0.64 | max|w| / 7 = 0.64 / 7 = 0.0914 |
- RMS error
- 0.028
- max 0.057
- Bits per weight
- 8
- 4 + 16 / 4 · groups of 128: 4.125
- 8 billion weights
- 8 GB
- 16 GB at 16-bit · 50%
Invented weights. Bits per weight assume a 16-bit scale per group and, for asymmetric groups, a zero point stored in as many bits as the codes; real formats store metadata differently.
What is actually quantized?
The Transformers overview describes methods and backend support. Distinguish three objects:
Four-bit weights can coexist with 16-bit activations and cache. Computation may locally dequantize weights before using floating-point kernels. W4A16 means 4-bit weights and 16-bit activations, not that every value in the process uses four bits. File containers, quantization algorithms, and execution kernels are also distinct. A downloadable format need not run efficiently on the chosen runtime and device.
Estimate capacity without hiding overhead
An invented dense model with exactly 8 billion weights needs 8 × 10^9 × 2 = 16 GB for raw 16-bit weights and 8 × 10^9 × 0.5 = 4 GB for raw 4-bit weights. These are decimal GB; in GiB they are about 14.9 GiB and 3.7 GiB.
Files also contain scales, zero points, unquantized tensors, and metadata. Running memory adds KV cache, scratch space, runtime overhead, and concurrent requests. Four gigabytes of weights do not imply that four gigabytes of VRAM support any context length. For mixture-of-experts models, distinguish activated parameters per token from weights stored or loaded; sparse activation does not remove the total model's storage requirement.
The suffix on a download names the format and bit width
Download a model locally and you often see one model listed as a column of files; the suffix on each name is its quantization. llama.cpp's quantize documentation gives the actual sizes of Llama 3.1 8B at each level:
The number in the suffix is the main bit width; the real average is always a little higher because the scales take space too. Per Hugging Face's GGUF notes, K means k-quant: weights are grouped into super-blocks of 256, split again into small blocks, each with its own scale. So Q4_K is about 4.5 bits in practice. The trailing S, M and L are levels of a mixture that keeps some tensors at higher precision, which is why Q4_K_M carries about 0.2 bits more than Q4_K_S. The IQ i-quants use an importance matrix (imatrix) to decide how to approximate, and the llama.cpp documentation says a suitable imatrix reduces the loss in perplexity and KL divergence. Check against the arithmetic in the previous section: 8 billion weights at 4 bits are 4 GB, yet the Q4_K_M file is 4.58 GiB, about 4.9 GB; the difference is that metadata and the higher-precision tensors.
Beyond GGUF, a few other formats are common:
- MLX: the format MLX uses on Apple Silicon. mlx-lm converts Hugging Face weights to a quantized version with
mlx_lm.convert -q; the documentation's example produces a 4-bit model. MLX files and GGUF are not interchangeable. - AWQ, GPTQ: GPU-quantized weights in safetensors, mainly for servers such as vLLM and Transformers. vLLM's quantization documentation lists hardware support for each. From vLLM 0.30.0, the
g_idxused by GPTQ activation ordering is ignored, so check older files that rely on it. - MXFP4: a 4-bit microscaling block floating-point format, also a GGUF type. gpt-oss-20b quantized its MoE weights to MXFP4 during post-training, and its card says it runs within 16 GB of memory.
- Vendor QAT files: Google publishes an official
qat-q4_0GGUF for Gemma 4 31B, a 17.7 GB file. The card says quantization-aware training preserves quality similar to bfloat16. It is a ready example of the QAT described in the next section.
KV-cache precision is not in the file name; you set it in the runtime. Ollama uses OLLAMA_KV_CACHE_TYPE, llama.cpp -ctk and -ctv, vLLM kv_cache_dtype. How much each level saves and what it costs is in the KV-cache note.
PTQ, QAT, and calibration data
Post-training quantization approximates existing weights. Quantization-aware training incorporates quantization effects into training. Rounding illustrates the principle, while more advanced methods reduce important errors using weight and activation distributions. GPTQ uses approximate second-order information for post-training weight quantization; AWQ uses activation statistics to protect important weight channels.
Another route is SmoothQuant: move part of the activation outliers into the weights so that both become easier to quantize. It serves server accelerators that quantize weights and activations together, unlike the weight-only AWQ.
Calibration data here estimates ranges or approximation errors; this differs from probability calibration. Code, long contexts, and multilingual tasks can have different activation distributions. Unrepresentative calibration text can leave a format strong on a broad benchmark but weak on the target job.
LoRA is a parameter-efficient adaptation method; distillation transfers teacher behavior to a student. Neither is numerical quantization, although they can be combined. See transfer learning, LoRA, and distillation.
Smaller does not guarantee faster
When one person generates token by token, every new token reads all the weights from GPU memory once, and reading takes far longer than computing; the inference-performance note works through this step with numbers. For the 8-billion-weight model above, 4 GB at 4-bit reads three quarters less per token than 16 GB at 16-bit, so decode gets faster. But if a low-bit kernel first unpacks the weights back to 16-bit before computing, does not suit the hardware, or falls back to a slow operator, the saved time is eaten again.
Long-prompt prefill computes many tokens at once, so its bottleneck is compute and reading fewer weight bytes helps little. Token-by-token decode and different batch sizes each have their own bottlenecks, so one speed number cannot summarize a quantization format.
Compare formats with the original model, runtime, hardware, prompts, and output budget fixed. Record peak memory, first-token latency, generation rate, task accuracy, and failure cases. Include long text, rare terms, structured outputs, and tool arguments rather than only perplexity or casual chat. If lower precision causes repeated retries, single-inference savings can disappear at the workflow level. Inference performance explains the measurement boundaries.