Attention Variants and KV-Cache Compression
How MHA, GQA and MLA set the size of the KV cache, and the context length, cache precision and prefix caching you can tune when running a model.
How MHA, GQA and MLA set the size of the KV cache, and the context length, cache precision and prefix caching you can tune when running a model.