Local deployment and inference

Local inference is a resource and serving problem. A weight file fitting in memory does not establish that a long context, multiple requests and the desired latency also fit.

Start with the single-device budget, then select a compatible runtime. Quantization explains what changes in the stored and computed numbers; KV Cache explains sequence-dependent memory. The performance article follows a request through loading, prefill and decoding.

Reading Order

Step	Article	What it explains
1	Single-Device Deployment: Weights, Memory and Context	A memory-first method for deciding which local model configuration may fit and remain useful on one consumer GPU.
2	Local Inference Runtimes	Choose a local runtime by artifact compatibility, hardware path, latency, throughput, observability, and security.
3	Model Quantization: Precision, Size, and Speed	Understand low-bit weight approximation, separate weight, activation, and KV-cache quantization, and estimate deployment costs.
4	Attention Variants and KV-Cache Compression	How MHA, GQA, MLA, cache quantization, and token-selection methods trade memory for implementation and quality risk.
5	Inference Performance: Loading, First Tokens, Decoding, and Throughput	Break requests into measurable stages and explain how concurrency, caching, and batching affect waiting time.

Use What You Read

Write down the model variant, precision, context, concurrency and hardware. Compare both task quality and cold/warm performance using the evaluation protocol; changing several settings at once hides the cause of a gain.

Return to the AI reading paths to choose a neighboring topic.

My reading · Opened notes are remembered in this browser.