4 docs tagged with "inference"

Break requests into measurable stages and explain how concurrency, caching, and batching affect waiting time.

A practical guide to choosing a local inference engine by model format, hardware fit (Apple Silicon included), context settings and security.

Follow a token through MoE routing to distinguish total parameters, active parameters, memory, speed, load balancing, and communication.

Check how to turn on speculative decoding in a local runtime, then work through probability correction and timing examples to see when it really speeds things up, and compare chains, trees, and self-speculation.