FlashAttention: Mathematical Foundations, IO-Aware Tiling, Online Softmax Scaling, and Memory-Hierarchy Optimization in Transformer Architectures
Standard multi-head self-attention represents the core computational bottleneck in scaling modern Transformer models to long context windows. While the theoretical arithmetic operations of self-attention scale quadratically with sequence length $O(N^2)$, modern GPU execution profiles reveal that standard implementations are constrained not by floating-point arithmetic throughput (FLOPs), but by memory access latency and bandwidth between High Bandwidth Memory (HBM) and on-chip Static Random-Acce
1 min
