Linear Attention2 articles

Linear Attention

Articles

Performers and FAVOR+: How Positive Orthogonal Random Features Linearize Transformer Attention
The quadratic complexity of standard self-attention has remained a central computational ceiling in Transformer architectures. Because standard attention computes pairwise similarity across all token pairs in a sequence of length $L$, memory consumption and compute scale as $O(L^2)$. For long contexts, high-resolution visual tokens, and biological sequence modeling, this quadratic bottleneck forces strict sequence truncation or aggressive hardware partitioning. In Rethinking Attention with Perf
August 23, 20261 min
Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck
Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ
August 19, 20261 min