Running locally

First ask whether to run locally at all: data that must stay on your machine and offline use are the strongest reasons. Many tiny calls are often given as a reason too, but hosted small tiers are already cheap, so price them with the Model & API Radar first. Once you decide to run locally, check in turn whether it fits, which model, which runtime, which settings and how fast; how the KV cache grows with context is covered in the How LLMs work group, in Attention Variants and KV-Cache Compression.

Reading Order

Step	Article	Your job
1	Single-Device Deployment: Weights, Memory and Context	Does it fit: budget one device’s memory for weights, context and headroom.
2	Local Models for Coding Agents	Which model: start from model-card facts to list coding models worth trying locally.
3	Local Inference Runtimes	Which runtime: choose an engine by model format, hardware, latency, throughput and security.
4	Model Quantization: Precision, Size, and Speed	Which settings: how low-bit numbers approximate weights, and what weight, activation and KV-cache quantization each save.
5	Inference Performance: Loading, First Tokens, Decoding, and Throughput	How fast: split a request into loading, prefill and decoding, and measure time to first token and throughput.
6	Speculative Decoding: Drafting, Verification, and Real Speed Gains	A little faster: a small model drafts, the large model verifies, and when that really speeds things up.

My reading