Ring Attention with Blockwise Transformers: Mathematical Foundations, Circular KV Passing, Overlapped Communication-Computation, and Linear Context Scaling
Scaling the context length of transformer-based large language models has historically collided with two fundamental walls: the $O(S^2)$ computational and memory complexity of standard self-attention, and the high-bandwidth memory (HBM) capacity of individual accelerator devices. While IO-aware tiling algorithms such as FlashAttention eliminate intermediate quadratic activation storage by computing softmax within fast on-chip SRAM, the entire sequence key and value tensors must still reside with
1 min