llms.blog
FlashAttention3 articles

FlashAttention

Articles

  • FlashAttention: Mathematical Foundations, IO-Aware Tiling, Online Softmax Scaling, and Memory-Hierarchy Optimization in Transformer Architectures

    Standard multi-head self-attention represents the core computational bottleneck in scaling modern Transformer models to long context windows. While the theoretical arithmetic operations of self-attention scale quadratically with sequence length $O(N^2)$, modern GPU execution profiles reveal that standard implementations are constrained not by floating-point arithmetic throughput (FLOPs), but by memory access latency and bandwidth between High Bandwidth Memory (HBM) and on-chip Static Random-Acce

    1 min

  • FlashAttention: IO-Aware Tiling, Online Softmax Mathematics, Memory Hierarchy Dynamics, and Kernel Evolution

    Standard multi-head attention is the computational core of transformer language models. While mathematically straightforward, standard attention implementations suffer from severe memory and throughput bottlenecks as sequence lengths grow. The bottleneck is not raw arithmetic compute, but memory traffic: repeatedly reading and writing intermediate attention matrices across GPU memory tiers. FlashAttention, introduced by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré in 202

    1 min

  • FlashAttention: Mathematical Foundations, Online Softmax Tiling, IO-Awareness, and Exact Attention Scaling

    Standard multi-head self-attention in the Transformer architecture exhibits quadratic time and memory complexity with respect to sequence length $N$. While the $O(N^2)$ computational complexity is widely cited, the primary performance bottleneck in production hardware is not arithmetic throughput, but memory access overhead. On modern GPU architectures such as NVIDIA A100 and H100, tensor processing cores execute matrix multiplications at teraflop and petaflop scales, but memory bandwidth betwee

    1 min