llms.blog
Attention11 articles

Attention

Articles

  • Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, KV-Cache Bandwidth Reduction, Uptraining Recipes, and Tensor Parallelism Implications

    Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, KV-Cache Bandwidth Reduction, Uptraining Recipes, and Tensor Parallelism Implications The KV-Cache Bandwidth Wall Autoregressive decoder inference is bottlenecked by memory bandwidth, not compute. At each decoding step, the model must reload the entire key-value (KV) cache from high-bandwidth memory (HBM) into the compute units. For a model with $H$ attention heads, sequence length $n$, head dimension $d_

    1 min

  • FlashAttention: IO-Aware Exact Attention, Tiling, Online Softmax, and the Evolution to FlashAttention-3

    FlashAttention: IO-Aware Exact Attention, Tiling, Online Softmax, and the Evolution to FlashAttention-3 The attention mechanism is the computational bottleneck of every Transformer model. Standard implementations materialize the full $N \times N$ attention matrix in high-bandwidth memory (HBM), incurring $O(N^2)$ memory reads and writes that dominate runtime long before arithmetic intensity saturates the GPU. FlashAttention and its successors eliminate this bottleneck by restructuring the atten

    1 min

  • Multi-Head Latent Attention: Low-Rank KV Compression, Decoupled RoPE, and Matrix Absorption

    Multi-Head Latent Attention: Low-Rank KV Compression, Decoupled RoPE, and Matrix Absorption Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2, addresses the KV cache bottleneck that has constrained long-context LLM inference. Rather than reducing the number of heads as in Grouped-Query Attention (GQA) or Multi-Query Attention (MQA), MLA compresses keys and values into a shared low-rank latent representation, caches that compressed form, and reconstructs full K and V matrices at use t

    1 min

  • FlashAttention: IO-Aware Tiling, Online Softmax Mathematics, Memory Hierarchy Dynamics, and Kernel Evolution

    Standard multi-head attention is the computational core of transformer language models. While mathematically straightforward, standard attention implementations suffer from severe memory and throughput bottlenecks as sequence lengths grow. The bottleneck is not raw arithmetic compute, but memory traffic: repeatedly reading and writing intermediate attention matrices across GPU memory tiers. FlashAttention, introduced by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré in 202

    1 min

  • Grouped-Query Attention and Multi-Query Attention: Mathematical Foundations, KV Cache Bandwidth Scaling, and Inference Economics

    Autoregressive generation in Large Language Models presents an asymmetric computational profile. During prompt processing (prefill), computation is compute-bound, executing large matrix multiplications that fully saturate modern Tensor Cores. During token-by-token generation (decoding), execution shifts entirely to a memory-bandwidth-bound regime. Each newly generated token requires fetching all prior Key and Value (KV) activation tensors from High Bandwidth Memory (HBM) to on-chip SRAM to compu

    1 min

  • FlashAttention: Mathematical Foundations, Online Softmax Tiling, IO-Awareness, and Exact Attention Scaling

    Standard multi-head self-attention in the Transformer architecture exhibits quadratic time and memory complexity with respect to sequence length $N$. While the $O(N^2)$ computational complexity is widely cited, the primary performance bottleneck in production hardware is not arithmetic throughput, but memory access overhead. On modern GPU architectures such as NVIDIA A100 and H100, tensor processing cores execute matrix multiplications at teraflop and petaflop scales, but memory bandwidth betwee

    1 min

  • In-Context Learning as Implicit Gradient Descent: How Transformers Optimize Models in Forward Activations

    When large language models (LLMs) adapt to new tasks from a handful of prompt demonstrations, their static weights remain completely untouched. No backpropagation runs through the network, no optimizer updates parameters, and no gradients are calculated. Yet, the model’s predictions improve steadily as more input-output examples are added to the prompt context. For years, this phenomenon (known as in-context learning, or ICL) was treated as an empirical black box. Recent theoretical and mechani

    1 min

  • Cross-Layer Attention: How Sharing Key-Value Tensors Across Transformer Layers Slashes KV Cache Memory

    In autoregressive large language models, the primary operational ceiling for serving long sequences and high batch concurrency is the Key-Value (KV) cache. During standard generation, every transformer layer computes and stores key and value activations for every token in the sequence to prevent quadratic recomputation during subsequent autoregressive decoding steps. While this mechanism transforms inference time complexity from $O(N^2)$ to $O(N)$ per generated token, it introduces a massive me

    1 min

  • Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces

    Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces The attention mechanism serves as the computational core of modern Transformer architectures, providing a mechanism for tokens to dynamically exchange information across arbitrary sequence positions without recurrent state transitions or fixed convolutional receptive fields. While single-head attention computes a single set of attention weights per token pair, modern large lan

    1 min

  • Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts

    Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and at

    1 min

  • Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    1 min