Attention Variants and KV-Cache Compression
How MHA, GQA and MLA set the size of the KV cache, and the context length, cache precision and prefix caching you can tune when running a model.
How MHA, GQA and MLA set the size of the KV cache, and the context length, cache precision and prefix caching you can tune when running a model.
Check how to turn on speculative decoding in a local runtime, then work through probability correction and timing examples to see when it really speeds things up, and compare chains, trees, and self-speculation.