Local Inference Runtimes

An HTTP 200 only proves that a request returned, not that the answer is right. A local model runs through at least four separable layers, and changing any of them can change quality, memory, latency, or tool behavior:

Before comparing runtimes, therefore, pin down what each layer uses; the compatibility gate below does exactly that.

Runtime Map

Runtime/product	Strong starting point	Main operator-owned risk
llama.cpp	GGUF, CPU/GPU portability, hybrid offload, low-level control	flags, templates, conversion provenance, and backend tuning
Ollama	managed local model packages and API	artifact/template drift; cloud models do not run on the local machine (see Security below)
LM Studio	desktop discovery, interactive comparison, local compatible API; GGUF through llama.cpp on all three systems, plus MLX models on Apple Silicon	GUI defaults and downloaded variants must be recorded
MLX / mlx-lm	on Apple Silicon the CPU and GPU share one memory pool, so weights are not copied; Ollama and LM Studio can both use it as a backend	Apple Silicon only; MLX files and GGUF are separate quantizations and not interchangeable
vLLM	CUDA/Python serving, continuous batching, throughput, compatible API	heavier environment, artifact support, scheduler/memory tuning

SGLang, Transformers, TensorRT-LLM, ROCm-specific, and vendor runtimes may be better for a measured need. The table is not an exhaustive rank.

Recent Changes That Affect the Gate

Checked on 2026-10-29 against each project's release notes. Only the three changes that alter how you install or call a runtime:

• Ollama 0.40.0 (September 25) runs model architectures that MLX supports on MLX automatically on Apple Silicon. The release notes name qwen3.8, gemma4 and qwen3.6 as examples. The latest release on October 28 is 0.40.13. The same model name can use different backends on a Mac and on an NVIDIA machine, so record which backend served each run.
• vLLM 0.32.0 (October 18) drops Python 3.10 and no longer reserves max_tokens from the context window. 0.31.0 (October 5) replaced online quantization through quantization="fp8" with the fp8_per_tensor shorthand, so old launch scripts need editing.
• SGLang 0.5.22 (October 19) requires Python 3.11 or newer.

Compatibility Gate

Before evaluation, freeze:

• exact model repository, revision, filename, and quantization;
• tokenizer, chat template, reasoning format/parser, and stop tokens;
• tool-call format and structured-output behavior;
• native/extended context, the context length actually set, KV-cache type, batch/concurrency, and GPU offload;
• runtime and driver versions, checked against the owner's hardware requirements for Ollama, LM Studio, or vLLM.

“The context length actually set” is the item most often missed. Ollama uses a 4096-token context window by default; a coding agent's system prompt plus a few files exceeds it, and whatever does not fit never reaches the model. Raise it with the OLLAMA_CONTEXT_LENGTH environment variable or num_ctx in the request, then check with the single-device budget that the cache fits. In llama.cpp, -c defaults to 0, which means the length stored in the model (server documentation).

OpenAI-compatible APIs standardize a useful subset of request shape. They do not guarantee identical reasoning fields, token accounting, tool IDs, streaming chunks, logprobs, cancellation, or errors.

Worked Choice

For one 16 GB consumer GPU or one Apple Silicon Mac and one interactive user:

1. start with a supported quantized artifact and native context;
2. use llama.cpp when exact GGUF/offload control matters, or Ollama/LM Studio for the quickest manual comparison. On a Mac, try the model's MLX build first, then compare the GGUF at the same size;
3. measure time to first token and decode speed at concurrency 1;
4. verify chat and tool formatting before judging intelligence;
5. move to vLLM only if its supported artifact and batching/serving benefits solve a real workload.

A 24 GB always-on multi-client service is a small-team need: throughput, queueing, cancellation and isolation outweigh desktop convenience. Choose a server with continuous batching such as vLLM or SGLang, and measure p95 time to first token and inter-token latency at fixed concurrency. Confirm that generation really stops when a client disconnects, and configure authentication and request-size limits before opening the port.

For speculative decoding, check draft–target compatibility before comparing latency, throughput, and memory at matched workload and quality.

Evaluation handoff

This page owns runtime compatibility, serving choices, and operational risk. The shared Local Model Evaluation protocol owns run cards, controlled factor changes, latency and throughput metrics, memory measurements, accepted-result rate, and repeated task results.

Security and Operations

For locality-sensitive work, distinguish the local server from where its selected model executes. Ollama cloud models are called in the CLI and desktop app by names such as gemma4:cloud; they need no download, and inference runs in Ollama's cloud. The documentation says Ollama processes cloud prompts and responses but does not use them to train models. When local-only execution is required, use only downloaded local models and disable cloud features. Set OLLAMA_NO_CLOUD=1, or put "disable_ollama_cloud": true in ~/.ollama/server.json. After restarting Ollama, the log should show Ollama cloud disabled: true. This disables Ollama cloud models and web search, not external tools in another harness; it is not a network sandbox.

• Network binding: Ollama binds 127.0.0.1:11434 by default; add authentication before changing the address with OLLAMA_HOST;
• Credentials: give the model server its own key, shared with no other service and kept out of any repository or log the harness can read;
• Telemetry: find out what usage data the runtime and client send outside, and switch off what you do not need;
• Resource limits: cap the size of each request, its context, and its output;
• Cancellation: after a client disconnects, confirm the server really stops generating and no longer holds the GPU;
• Supply chain: audit model files and templates as supply-chain inputs.

Do not let a compatible endpoint become an unaudited internal network proxy.

My reading