Multi-Head Latent Attention: Low-Rank KV Compression, Decoupled RoPE, and Matrix Absorption
Multi-Head Latent Attention: Low-Rank KV Compression, Decoupled RoPE, and Matrix Absorption Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2, addresses the KV cache bottleneck that has constrained long-context LLM inference. Rather than reducing the number of heads as in Grouped-Query Attention (GQA) or Multi-Query Attention (MQA), MLA compresses keys and values into a shared low-rank latent representation, caches that compressed form, and reconstructs full K and V matrices at use t
1 min