MQA

Multi-Query Attention: How Single Key-Value Head Sharing Slashed Transformer Serving Bottlenecks

Multi-query attention (MQA) is an architectural modification to the Transformer attention mechanism designed to resolve the memory bandwidth bottleneck during autoregressive token generation. First proposed by Noam Shazeer in the 2019 paper Fast Transformer Decoding: One Write-Head is All You Need, MQA alters the ratio of query, key, and value heads by sharing a single key head and a single value head across all query heads in each Transformer layer. While standard multi-head attention (MHA) pr