Single-GPU Local Models
“Runs on one GPU” can mean weights load with CPU offload, one short request completes, or a useful context runs at interactive speed. These are different claims.
Memory Budget
Raw weight memory starts with:
weight GiB ≈ parameters × bits per weight ÷ 8 ÷ 2^30
A nominal 7B model at 4 bits is about 3.3 GiB of raw weights; a nominal 24B model is about 11.2 GiB. Real artifacts add quantization metadata, embeddings or unquantized tensors, allocator/workspace overhead, and runtime buffers.
KV-cache memory is architecture-dependent but grows approximately with layers, retained KV heads/head dimension, bytes per cache element, batch, and sequence length:
KV bytes ≈ 2 × layers × KV width × sequence × bytes × batch
Always use runtime measurements or an architecture-aware calculator for the actual model. Reserve output tokens and concurrent requests before declaring a fit.
For MoE models, active parameters estimate compute per token; the total experts still contribute to weight storage unless a specialized offload scheme says otherwise.
Starting Tiers, Not Guarantees
| Available VRAM | Reasonable first trial | Likely compromise |
|---|---|---|
| 8 GB | 3–4B at 4/5-bit; selected 7B at short context | capacity, context, or CPU offload |
| 12 GB | 7–8B quantized; cautious 12–14B experiments | context and speed headroom |
| 16 GB | 12–14B quantized; supported gpt-oss-20b MXFP4 path | template/KV/runtime constraints |
| 24 GB | 20–24B quantized; selected 30B MoE artifacts | long-context and concurrency pressure |
CPU offload expands placement, not VRAM. It may reduce throughput enough to make an autonomous loop impractical.
Candidate Facts
- Qwen3-4B: 4B, native 32K, thinking/non-thinking control, broad runtime support; the card warns against greedy decoding in thinking mode because of degradation/repetition.
- Phi-4-mini-instruct: 3.8B and advertised 128K context for constrained use; its card explicitly notes limited factual capacity and stronger Python coverage.
- gpt-oss-20b: 21B total/3.6B active; OpenAI states the MXFP4 configuration runs within 16 GB of memory and requires Harmony formatting.
- Mistral Small 3.1 24B: its card says a quantized build can fit an RTX 4090 or 32 GB Mac, while BF16 serving needs roughly 55 GB. Precision is part of the claim.
These facts create candidates, not a quality ranking.
Worked Fit Procedure
- Define task, concurrency, context, output reserve, and latency target.
- Pick an exact artifact and compute raw weight size.
- verify template and runtime support before download if possible.
- Start at native/short context and concurrency 1.
- Measure peak VRAM/RAM, offload, first-token latency, decode speed, and accepted result.
- Increase context independently; stop before OOM or unacceptable latency.
- Keep 10–20% operational headroom rather than tuning to one lucky peak.
Fit Versus Usefulness
A smaller model that reliably emits valid tools may outperform a larger offloaded model in an agent loop. A larger context that lowers speed and attention quality may be worse than retrieval into a shorter window. Decide with the local evaluation protocol, not parameter count.
Bias and Safety
These tiers are shaped by consumer NVIDIA-style VRAM and common quantizations. Apple unified memory, AMD, integrated GPUs, NPUs, power limits, and server accelerators behave differently. Model-card memory claims may use a favorable runtime and context. Bind local servers to loopback, review licenses and telemetry, and do not equate local execution with automatic privacy or safety.