Single-Device Deployment: Weights, Memory and Context

“Runs on one GPU” can mean that the weights barely load, that a short request finishes with CPU offload, or that a useful context runs at interactive speed. To decide whether a configuration is practical, check the memory, context length, and speed your task needs. An interactive coding session and an overnight batch can tolerate very different delays.

This page owns the capacity question: what must fit on a device and what changes as context and concurrency grow. The dated list of candidate models lives only in coding-model selection; for numeric formats and measured speed continue with quantization and inference performance.

Start with the memory budget

Raw weight memory is approximately

$$\text{weight bytes}\approx\frac{\text{parameter count}\times\text{bits per weight}}{8}.$$

A 27B model at four bits is about 12.6 GiB before quantization metadata, tensors kept at higher precision, runtime workspaces, and the KV cache. Artifact size is a better starting point than parameter arithmetic when the exact quantization already exists.

For conventional MHA or GQA models, one sequence's KV cache is approximately

$$2\times L\times S\times d_{kv}\times b,$$

where the factor 2 counts one Key and one Value, $L$ is layer count, $S$ is retained sequence length, $d_{kv}$ is cached key/value width per layer, and $b$ is bytes per element.

Plug in Llama 3 8B's shape: 32 layers, 8 key/value heads of 128 dimensions each, so a key/value width of 1024, stored in BF16 at 2 bytes. One token takes 2×32×1024×2 = 131,072 bytes, or 128 KiB. At a 32K-token context that is 4 GiB; at 128K it is 16 GiB, about as large as all of the model's BF16 weights (8 billion × 2 bytes = 16 GB). A published context limit therefore does not mean the full window fits on one GPU, runs quickly, or retains enough quality for the task. The KV-cache note works through more cases.

The $b$ in the formula is a setting you control. Ollama sets cache precision with OLLAMA_KV_CACHE_TYPE; its FAQ says q8_0 uses about half the memory of the default f16 and q4_0 about a quarter, provided Flash Attention is on. The 16 GiB cache at 128K above drops to about 8 GiB with q8_0; q4_0 saves more and costs more quality.

MLA and other attention designs use different cache layouts. Batch size and parallel sequences multiply the requirement.

Leave room for output tokens, temporary buffers, the display driver, and ordinary runtime variation. A configuration that survives one prompt with almost no free VRAM is not yet a useful capacity plan.

Reasonable first experiments

The table assumes 4-bit weights, one sequence, and a context of about 8K tokens. Architecture, quantization format, backend, cache precision, and GPU support all move the boundary; with another format or a longer context, go back to the formulas above. A model name cannot stand in for that step: Mistral's Small 4 has 119B parameters, and “Small” says nothing about fitting a consumer card.

Available VRAM	First trial	Likely pressure point
8 GB	3–4B quantized; selected 7–8B models at short context	weights, context, and offload
12 GB	7–8B quantized; cautious larger-model experiments	KV cache and runtime workspace
16 GB	12–14B at comfortable quantization, or a tightly fitted 20–27B artifact	context, concurrency, and headroom
24 GB	20–27B quantized with more cache room; selected larger or MoE artifacts	long context and multiple sequences

On a Mac, unified memory is one shared budget

Apple Silicon has no separate VRAM. In the words of the MLX documentation, the CPU and GPU have direct access to the same memory pool, so weights are not copied between them. The cost is that the model shares that memory with the system, the browser and the editor: a 16 GB Mac leaves the model far less than 16 GB.

Metal gives a reference figure for how much the GPU can use. recommendedMaxWorkingSetSize is an approximation of how many bytes the GPU can allocate without affecting its runtime performance; keeping resources below it keeps the GPU fast. mlx-lm warns that models large relative to total RAM can be slow. When the model fits, macOS 15 and later can raise the wired-memory limit with sudo sysctl iogpu.wired_limit_mb=N, where N is larger than the model in megabytes and smaller than the machine's memory.

Speed then depends on memory bandwidth. On Apple's MacBook Pro specifications, the 16 GB M6 has 153 GB/s. The M5 Pro has 307 GB/s and the M5 Max with a 40-core GPU 614 GB/s. The same 4-bit file decodes on these three machines with ceilings roughly in proportion to bandwidth; the arithmetic is in inference performance.

This table keeps the previous table's assumptions, after the system and everyday apps have taken their share:

Unified memory	Roughly comparable to	Likely pressure point
16 GB	an 8–12 GB card: 7–8B quantized, short context	the system and other apps compete for memory
24–32 GB	a 16 GB card: 12–14B, or a tightly fitted 20–27B artifact	context and headroom
48–64 GB	a 24 GB card or more: 20–32B with long context, some MoE	bandwidth sets speed
128 GB	larger MoE or 70B-class quantized artifacts	bandwidth and prefill time

On a Mac the same model often comes as both GGUF and MLX files, with different quantization, size and speed; runtime selection covers which runtime reads which.

Offload is a trade, not a failure

CPU or system-memory offload can make a larger model run, but it moves data across a much slower path and may reduce decode speed sharply. There is no universal tokens-per-second cliff: the penalty depends on the model, offloaded layers, memory bandwidth, PCIe or other interconnect, backend, prompt length, and batch size. The same offload optimization can pay off several times more on one machine than another. ExLlamaV3 sped up CPU offload in several October releases; after an upgrade or on a new machine, re-measure with your own workload.

Full GPU residency is desirable for latency-sensitive work, not a rule for every workload. An overnight batch may tolerate offload that would make an interactive coding loop unpleasant.

Fit procedure

1. Define the task, concurrency, input length, output reserve, and acceptable latency.
2. Choose an exact artifact, template, and runtime.
3. Start with a short context and one sequence.
4. Record peak GPU/RAM use, offload, time to first token, decode speed, and task result.
5. Increase context separately from concurrency so the source of a failure is visible.
6. Test the longest information dependency the real workload needs, not only a prompt full of filler.
7. Keep operational headroom and record the configuration that was actually tested.

A smaller resident model can beat a larger offloaded one in a tool loop. A larger context can also make the system slower without making it better at finding the right evidence. Use the local evaluation protocol to decide with tasks rather than parameter count.

The Accelerate model memory estimator guide shows how to estimate parameter memory for different numeric types without loading all model weights. Compare that estimate with the weight arithmetic above, then measure the complete runtime separately: cache, activations, and temporary buffers still need their own budget.

My reading