Attention11 articles

Attention

Articles

Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, KV-Cache Bandwidth Reduction, Uptraining Recipes, and Tensor Parallelism Implications
Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, KV-Cache Bandwidth Reduction, Uptraining Recipes, and Tensor Parallelism Implications The KV-Cache Bandwidth Wall Autoregressive decoder inference is bottlenecked by memory bandwidth, not compute. At each decoding step, the model must reload the entire key-value (KV) cache from high-bandwidth memory (HBM) into the compute units. For a model with $H$ attention heads, sequence length $n$, head dimension $d_
August 28, 20261 min
FlashAttention: IO-Aware Exact Attention, Tiling, Online Softmax, and the Evolution to FlashAttention-3
FlashAttention: IO-Aware Exact Attention, Tiling, Online Softmax, and the Evolution to FlashAttention-3 The attention mechanism is the computational bottleneck of every Transformer model. Standard implementations materialize the full $N \times N$ attention matrix in high-bandwidth memory (HBM), incurring $O(N^2)$ memory reads and writes that dominate runtime long before arithmetic intensity saturates the GPU. FlashAttention and its successors eliminate this bottleneck by restructuring the atten
August 28, 20261 min
Multi-Head Latent Attention: Low-Rank KV Compression, Decoupled RoPE, and Matrix Absorption
Multi-Head Latent Attention: Low-Rank KV Compression, Decoupled RoPE, and Matrix Absorption Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2, addresses the KV cache bottleneck that has constrained long-context LLM inference. Rather than reducing the number of heads as in Grouped-Query Attention (GQA) or Multi-Query Attention (MQA), MLA compresses keys and values into a shared low-rank latent representation, caches that compressed form, and reconstructs full K and V matrices at use t
August 26, 20261 min
FlashAttention: IO-Aware Tiling, Online Softmax Mathematics, Memory Hierarchy Dynamics, and Kernel Evolution
Standard multi-head attention is the computational core of transformer language models. While mathematically straightforward, standard attention implementations suffer from severe memory and throughput bottlenecks as sequence lengths grow. The bottleneck is not raw arithmetic compute, but memory traffic: repeatedly reading and writing intermediate attention matrices across GPU memory tiers. FlashAttention, introduced by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré in 202
August 26, 20261 min
Grouped-Query Attention and Multi-Query Attention: Mathematical Foundations, KV Cache Bandwidth Scaling, and Inference Economics
Autoregressive generation in Large Language Models presents an asymmetric computational profile. During prompt processing (prefill), computation is compute-bound, executing large matrix multiplications that fully saturate modern Tensor Cores. During token-by-token generation (decoding), execution shifts entirely to a memory-bandwidth-bound regime. Each newly generated token requires fetching all prior Key and Value (KV) activation tensors from High Bandwidth Memory (HBM) to on-chip SRAM to compu
August 26, 20261 min
FlashAttention: Mathematical Foundations, Online Softmax Tiling, IO-Awareness, and Exact Attention Scaling
Standard multi-head self-attention in the Transformer architecture exhibits quadratic time and memory complexity with respect to sequence length $N$. While the $O(N^2)$ computational complexity is widely cited, the primary performance bottleneck in production hardware is not arithmetic throughput, but memory access overhead. On modern GPU architectures such as NVIDIA A100 and H100, tensor processing cores execute matrix multiplications at teraflop and petaflop scales, but memory bandwidth betwee
August 26, 20261 min
In-Context Learning as Implicit Gradient Descent: How Transformers Optimize Models in Forward Activations
When large language models (LLMs) adapt to new tasks from a handful of prompt demonstrations, their static weights remain completely untouched. No backpropagation runs through the network, no optimizer updates parameters, and no gradients are calculated. Yet, the model’s predictions improve steadily as more input-output examples are added to the prompt context. For years, this phenomenon (known as in-context learning, or ICL) was treated as an empirical black box. Recent theoretical and mechani
August 22, 20261 min
Cross-Layer Attention: How Sharing Key-Value Tensors Across Transformer Layers Slashes KV Cache Memory
In autoregressive large language models, the primary operational ceiling for serving long sequences and high batch concurrency is the Key-Value (KV) cache. During standard generation, every transformer layer computes and stores key and value activations for every token in the sequence to prevent quadratic recomputation during subsequent autoregressive decoding steps. While this mechanism transforms inference time complexity from $O(N^2)$ to $O(N)$ per generated token, it introduces a massive me
August 21, 20261 min
Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces
Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces The attention mechanism serves as the computational core of modern Transformer architectures, providing a mechanism for tokens to dynamically exchange information across arbitrary sequence positions without recurrent state transitions or fixed convolutional receptive fields. While single-head attention computes a single set of attention weights per token pair, modern large lan
August 20, 20261 min
Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts
Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and at
August 20, 20261 min
Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences
Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent
August 18, 20261 min