RingAttention and Context Parallelism: Mathematical Foundations, Block-Wise Exact Attention Over Ring Topologies, Overlapped Peer-to-Peer Communication, and Infinite-Context Scaling

Scaling context windows from thousands to millions of tokens has transformed large language models from localized sequence processors into comprehensive repository analyzers, long-form document synthesizers, and multi-modal world simulators. However, standard self-attention exhibits quadratic computational and memory complexity with respect to sequence length $S$. While IO-aware memory optimizations like FlashAttention eliminate intermediate attention matrix materialization in High Bandwidth Memory (HBM), the per-device key-value (KV) activations and query tensors for a multi-million-token sequence rapidly exceed the physical memory capacity of any single accelerator.

Distributed sequence scaling requires partitioning the sequence dimension across clusters of GPUs. Traditional distributed training paradigms—such as Tensor Parallelism (TP) and Pipeline Parallelism (PP)—cannot scale context length indefinitely. Tensor Parallelism partitions model weights and activations along hidden and head dimensions, bounding the maximum parallel degree by the number of attention heads (typically 32 to 128) while imposing high all-reduce communication overheads. Pipeline Parallelism partitions layers across devices, leaving the per-device activation memory for long sequences unchanged.

Context Parallelism (CP) addresses this fundamental limitation by sharding the sequence dimension $S$ across $N$ devices. Among context-parallel paradigms, RingAttention, introduced by Liu et al. (2023) building upon Blockwise Parallel Transformers (Liu & Abbeel, 2023), organizes GPUs into a logical 1D ring network. By circulating Key-Value blocks asynchronously along the ring while computing blockwise attention with running online softmax normalizers, RingAttention achieves exact attention computation with $O(S/N)$ per-device memory scaling and near-complete communication-compute overlap.

+---------------------------------------------------------------------------------------------------+
|                           DISTRIBUTED SEQUENCE PARALLELISM SPECTRUM                              |
+---------------------------------------------------------------------------------------------------+
| Paradigm           | Sharding Dimension | Communication Type       | Max Scaling Limit            |
+--------------------+--------------------+--------------------------+------------------------------+
| Tensor Parallelism | Head / Hidden ($d$) | Intra-node All-Reduce    | Bound by Attention Heads $H$ |
| DeepSpeed-Ulysses  | Sequence ($S$)     | All-to-All (Heads <-> S) | Bound by Attention Heads $H$ |
| RingAttention      | Sequence ($S$)     | Ring P2P Asynchronous   | Arbitrary Cluster Scale $N$  |
| 2D Hybrid (USP)    | Sequence ($S$)     | Intra-All2All + Inter-P2P| Massive Hybrid Scale ($N*M$) |
+--------------------+--------------------+--------------------------+------------------------------+

1. The Sequence Length Bottleneck in Distributed Systems

Standard Multi-Head Attention (MHA) projects an input sequence $X \in \mathbb{R}^{S \times d}$ into Query ($Q$), Key ($K$), and Value ($V$) tensors:

$$Q = X W^Q, \quad K = X W^K, \quad V = X W^V$$

where $W^Q, W^K, W^V \in \mathbb{R}^{d \times d}$, $S$ is sequence length, and $d$ is hidden dimension. For $H$ attention heads with head dimension $d_k = d/H$, attention output for a single head is evaluated as:

$$A = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}} + M\right) V$$

where $M \in \mathbb{R}^{S \times S}$ denotes an optional causal mask.

Memory and Activation Scaling

In standard backpropagation, calculating gradients with respect to $Q$, $K$, and $V$ requires storing intermediate activations. For a sequence of length $S = 1{,}000{,}000$ with $L = 32$ layers, $H = 32$ heads, and $d_k = 128$ in 16-bit precision ($\text{sizeof}(\text{fp16}) = 2$ bytes):

1. KV Activation Footprint per Layer:

$$\text{Memory}_{\text{KV}} = 2 \times S \times d \times 2 \text{ bytes} = 2 \times 10^6 \times 4096 \times 2 \approx 16.38 \text{ GB}$$

1. Total Model Activation Footprint across 32 Layers:

$$\text{Memory}_{\text{Total KV}} = 32 \times 16.38 \text{ GB} \approx 524.2 \text{ GB}$$

Even before accounting for attention score logits, optimizer states, and model parameters, a 1-million-token sequence exceeds the 80 GB or 141 GB HBM capacity of modern NVIDIA H100 or H200 accelerators by nearly an order of magnitude.

Failure Modes of Standard 3D Parallelism for Long Contexts

• Data Parallelism (DDP / FSDP / ZeRO-3): Replicates or shards weights across GPUs, but each GPU must still execute the forward and backward pass for a full sequence slice of length $S$. Per-device activation memory remains $O(S)$.
• Tensor Parallelism (Megatron-LM): Splits $W^Q, W^K, W^V$ along head or column dimensions. The sequence dimension $S$ remains unpartitioned within each GPU. Furthermore, the maximum TP degree is physically capped by $H$ ($TP \le 32$ or $64$). Beyond single-node NVLink domains, TP communication latency degrades throughput catastrophically.
• Pipeline Parallelism (PP): Distributes layers $1 \dots L$ across pipeline stages. Activations for sequence $S$ must still reside in stage memory, creating severe bubble overheads ($F_{\text{bubble}} = \frac{p-1}{m+p-1}$) when microbatch counts $m$ are small due to memory constraints.

Context Parallelism is therefore essential: it shards the sequence length $S$ into $N$ equal chunks of size $B = S/N$, distributing the memory burden such that each device stores only $O(S/N)$ tokens.

2. Mathematical Foundations: Exact Online Softmax Across Distributed Blocks

The core mathematical enabler of blockwise attention without full sequence materialization is the Online Softmax algorithm (Milakov & Gimelshein, 2018; Dao et al., 2022). RingAttention generalizes online softmax from single-GPU SRAM tiling to multi-node distributed network topologies.

The Classical Online Softmax Formulation

Given a query vector $q \in \mathbb{R}^{1 \times d_k}$ and key vectors partitioned into $T$ sequential blocks $K = [K^{(1)}; K^{(2)}; \dots; K^{(T)}]$ with $K^{(i)} \in \mathbb{R}^{B \times d_k}$, we define intermediate pre-softmax logits for block $i$ as:

$$S^{(i)} = \frac{q (K^{(i)})^T}{\sqrt{d_k}} \in \mathbb{R}^{1 \times B}$$

To avoid numerical overflow when computing $\exp(S^{(i)})$, standard softmax subtracts the global maximum $m = \max_j S_j$. In a sequential streaming setting, the running maximum $m^{(i)}$ and running denominator normalizer $l^{(i)}$ are updated iteratively:

$$m^{(i)} = \max\left(m^{(i-1)}, \max_{j} S^{(i)}_j\right)$$

$$\tilde{P}^{(i)} = \exp\left(S^{(i)} - m^{(i)}\right)$$

$$l^{(i)} = \exp\left(m^{(i-1)} - m^{(i)}\right) l^{(i-1)} + \sum_{j=1}^B \tilde{P}^{(i)}_j$$

The unnormalized output accumulator $O^{(i)} \in \mathbb{R}^{1 \times d_v}$ is updated by rescaling the previous accumulator and adding the contribution of the current Value block $V^{(i)} \in \mathbb{R}^{B \times d_v}$:

$$O^{(i)} = \exp\left(m^{(i-1)} - m^{(i)}\right) O^{(i-1)} + \tilde{P}^{(i)} V^{(i)}$$

After processing all $T$ blocks, the exact attention output vector is recovered via normalization:

$$A = \frac{O^{(T)}}{l^{(T)}}$$

This decomposition is mathematically identical to computing softmax over the fully concatenated sequence:

$$\frac{O^{(T)}}{l^{(T)}} = \frac{\sum_{i=1}^T \exp(S^{(i)} - m^{(T)}) V^{(i)}}{\sum_{i=1}^T \exp(S^{(i)} - m^{(T)})} = \text{Softmax}\left(\frac{q K^T}{\sqrt{d_k}}\right) V$$

+---------------------------------------------------------------------------------------------------+
|                        ONLINE SOFTMAX STATE TRANSITION STEP                                       |
+---------------------------------------------------------------------------------------------------+
|  Previous State: (m^(i-1), l^(i-1), O^(i-1))                                                      |
|                                                                                                   |
|  1. Compute Local Block Logits:       S^(i) = q (K^(i))^T / sqrt(d_k)                             |
|  2. Update Running Maximum:           m^(i) = max(m^(i-1), max(S^(i)))                            |
|  3. Compute Rescaled Exponentials:    P_tilde^(i) = exp(S^(i) - m^(i))                            |
|  4. Rescale & Accumulate Normalizer:  l^(i) = exp(m^(i-1) - m^(i)) * l^(i-1) + sum(P_tilde^(i))  |
|  5. Rescale & Accumulate Output:      O^(i) = exp(m^(i-1) - m^(i)) * O^(i-1) + P_tilde^(i)*V^(i)  |
|                                                                                                   |
|  Final Normalized Output:             A = O^(T) / l^(T)                                           |
+---------------------------------------------------------------------------------------------------+

3. Distributed Ring Topology and Overlapped Communication

RingAttention maps this blockwise recurrence onto a 1D logical ring of $N$ physical devices (indexed $k \in \{0, 1, \dots, N-1\}$).

Initial Sequence Allocation

The full sequence of length $S$ is partitioned into $N$ contiguous blocks of size $B = S/N$:

$$X = [X_0, X_1, \dots, X_{N-1}], \quad X_k \in \mathbb{R}^{B \times d}$$

Each GPU $k$ computes and permanently retains its local Query block $Q_k = X_k W^Q$, while initializing its local Key and Value buffers with $K_k^{(0)} = X_k W^K$ and $V_k^{(0)} = X_k W^V$. Each GPU also initializes its local online softmax statistics:

$$m_k^{(0)} = -\infty, \quad l_k^{(0)} = 0, \quad O_k^{(0)} = 0$$

+---------------------------------------------------------------------------------------------------+
|                            RINGATTENTION 1D LOGICAL RING ROTATION                                 |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
|             [ GPU 0 ]  ---- Send (K_0, V_0) --->  [ GPU 1 ]                                       |
|             (Holds Q_0)                           (Holds Q_1)                                     |
|                 ^                                     |                                           |
|                 |                                     |                                           |
|            Send (K_3, V_3)                       Send (K_1, V_1)                                  |
|                 |                                     |                                           |
|                 |                                     v                                           |
|             [ GPU 3 ]  <--- Send (K_2, V_2) ----  [ GPU 2 ]                                       |
|             (Holds Q_3)                           (Holds Q_2)                                     |
|                                                                                                   |
|  Step t=0: Local Attention (Q_k, K_k, V_k)                                                        |
|  Step t=1: Rotated Attention (Q_k, K_(k-1), V_(k-1))                                              |
|  Step t=2: Rotated Attention (Q_k, K_(k-2), V_(k-2))                                              |
|  Step t=3: Rotated Attention (Q_k, K_(k-3), V_(k-3))                                              |
+---------------------------------------------------------------------------------------------------+

The $N$-Step Ring Schedule

The computation executes across $N$ discrete steps $t \in \{0, 1, \dots, N-1\}$:

1. Local Computation: In step $t$, GPU $k$ holds the key-value block corresponding to rank $(k - t) \bmod N$. It evaluates blockwise attention between static local queries $Q_k$ and current key-value pair $(K^{(t)}_{\text{curr}}, V^{(t)}_{\text{curr}})$ using an optimized FlashAttention kernel, updating local state $(m_k, l_k, O_k)$.
2. Asynchronous Peer-to-Peer Transfer: Concurrently with step 1 computation, GPU $k$ issues non-blocking peer-to-peer (P2P) transfers:

• Asynchronously sends $(K^{(t)}_{\text{curr}}, V^{(t)}_{\text{curr}})$ to its downstream neighbor $(k + 1) \bmod N$.
• Asynchronously receives $(K^{(t+1)}_{\text{next}}, V^{(t+1)}_{\text{next}})$ from its upstream neighbor $(k - 1) \bmod N$.

1. Synchronization & State Swap: Once both local GEMM execution and P2P communication complete, GPU $k$ swaps buffers and advances to step $t+1$.

At the end of $N$ ring steps, key-value blocks have traversed the complete ring and returned to their origin devices. Each GPU normalizes its local accumulator $A_k = O_k^{(N)} / l_k^{(N)}$, producing the exact mathematical output corresponding to full sequence attention.

4. Communication vs. Compute Roofline Overlap Analysis

The core efficiency requirement of RingAttention is hiding communication latency behind block computation time ($T_{\text{comm}} \le T_{\text{comp}}$).

Arithmetic and Communication Formulations

For a sequence chunk of size $B = S/N$, hidden dimension $d$, and number of heads $H$:

1. Computation per Ring Step:

Evaluating attention between $Q \in \mathbb{R}^{B \times d}$ and $K, V \in \mathbb{R}^{B \times d}$ requires two matrix multiplications ($Q K^T$ and $P V$): $$\text{FLOPs}_{\text{step}} = 4 B^2 d$$ Given a GPU with peak dense compute throughput $C_{\text{peak}}$ (e.g., 989 TFLOP/s FP16/BF16 on NVIDIA H100 SXM5) and kernel execution efficiency $\eta_{\text{comp}} \approx 0.60$: $$T_{\text{comp}} = \frac{4 B^2 d}{\eta_{\text{comp}} C_{\text{peak}}}$$

1. Communication per Ring Step:

Each step transfers key and value blocks $K, V \in \mathbb{R}^{B \times d}$ in 16-bit precision ($2$ bytes per element): $$\text{Bytes}_{\text{step}} = 2 \times (2 B d) = 4 B d \text{ bytes}$$ Given bidirectional inter-GPU interconnect bandwidth $W_{\text{net}}$ (e.g., 900 GB/s for NVLink 4, or 50 GB/s for 400 Gbps InfiniBand) with communication efficiency $\eta_{\text{comm}} \approx 0.85$: $$T_{\text{comm}} = \frac{4 B d}{\eta_{\text{comm}} W_{\text{net}}}$$

Overlap Condition and Critical Block Size

To achieve 100% communication overlap ($T_{\text{comm}} \le T_{\text{comp}}$), the per-device block size $B$ must satisfy:

$$\frac{4 B d}{\eta_{\text{comm}} W_{\text{net}}} \le \frac{4 B^2 d}{\eta_{\text{comp}} C_{\text{peak}}} \implies B \ge \frac{\eta_{\text{comp}} C_{\text{peak}}}{\eta_{\text{comm}} W_{\text{net}}}$$

+---------------------------------------------------------------------------------------------------+
|                    CRITICAL BLOCK SIZE (B_crit) FOR FULL OVERLAP                                  |
+---------------------------------------------------------------------------------------------------+
| Hardware Interconnect    | Peak FP16 FLOPs | Net Bandwidth | Critical Block Size (B_crit)         |
+--------------------------+-----------------+---------------+--------------------------------------+
| Intra-Node NVLink 4      | 989 TFLOP/s     | 900 GB/s      | ~775 tokens                          |
| Inter-Node InfiniBand    | 989 TFLOP/s     | 50 GB/s (400G)| ~13,960 tokens                       |
| Inter-Node Multi-Rail IB | 989 TFLOP/s     | 400 GB/s (8x) | ~1,745 tokens                        |
+--------------------------+-----------------+---------------+--------------------------------------+

In typical multi-node clusters with 8x 400 Gbps InfiniBand rails (3.2 Tbps aggregate bandwidth), the critical block size $B_{\text{crit}}$ is approximately 1,745 tokens. Because long-context training tasks operate with per-device sequence chunks of $B = 8{,}192$ to $65{,}536$ tokens, $T_{\text{comp}} \gg T_{\text{comm}}$, ensuring that communication is fully hidden behind compute.

5. Causal Masking and Load Balancing: Striped vs. Zigzag Ring Attention

In autoregressive language models, attention is strictly causal: token $i$ can only attend to tokens $j \le i$, producing a lower-triangular attention matrix.

+---------------------------------------------------------------------------------------------------+
|                             CAUSAL ATTENTION MATRIX LOAD IMBALANCE                                |
+---------------------------------------------------------------------------------------------------+
|            Block 0     Block 1     Block 2     Block 3                                            |
|  Block 0 [   /\    ] [         ] [         ] [         ]  <- GPU 0 (Computes 1 block, idles 3)    |
|  Block 1 [  Full   ] [   /\    ] [         ] [         ]  <- GPU 1 (Computes 2 blocks, idles 2)   |
|  Block 2 [  Full   ] [  Full   ] [   /\    ] [         ]  <- GPU 2 (Computes 3 blocks, idles 1)   |
|  Block 3 [  Full   ] [  Full   ] [  Full   ] [   /\    ]  <- GPU 3 (Computes 4 blocks, idles 0)   |
+---------------------------------------------------------------------------------------------------+

The Causal Bubble Problem

In naive sequential partitioning, GPU $k$ holds queries for tokens in range $[k \cdot B, (k+1) \cdot B - 1]$. When key-value blocks from ranks $j > k$ arrive, they reside entirely in the masked upper-triangular region ($j > i$), requiring zero computation. Consequently:

• GPU 0 computes only 1 step and idles for $N-1$ steps.
• GPU $N-1$ computes for all $N$ steps.
• Total Compute Efficiency: Exactly $\frac{N(N+1)/2}{N^2} \approx 50\%$, wasting half of the cluster's processing power.

Solution 1: Striped Attention

Brandon et al. (2024) proposed Striped Attention, which shards tokens across GPUs round-robin rather than contiguously:

$$\text{GPU}_k \text{ receives tokens } \{i \mid i \equiv k \pmod N\}$$

Because each GPU holds an identical distribution of early, middle, and late tokens across the entire sequence length $S$, the causal mask ratio for every GPU at every ring step is identical ($\approx 50\%$). While Striped Attention perfectly eliminates load imbalance, it introduces token permutation overheads for subsequent operations like Rotary Position Embeddings (RoPE) and LayerNorm.

Solution 2: Zigzag Ring Attention

Zhang et al. (2024) introduced Zigzag Ring Attention, which preserves contiguous local chunks without token permutation.

+---------------------------------------------------------------------------------------------------+
|                                 ZIGZAG SEQUENCE ALLOCATION                                        |
+---------------------------------------------------------------------------------------------------+
|  Sequence is divided into 2N chunks: [c_0, c_1, ..., c_(2N-1)]                                    |
|                                                                                                   |
|  GPU 0 holds: [ c_0 , c_7 ]   (First chunk + Last chunk)                                          |
|  GPU 1 holds: [ c_1 , c_6 ]                                                                       |
|  GPU 2 holds: [ c_2 , c_5 ]                                                                       |
|  GPU 3 holds: [ c_3 , c_4 ]   (Middle chunks)                                                     |
+---------------------------------------------------------------------------------------------------+

Each GPU $k$ is assigned two blocks: one from the front of the sequence ($c_k$) and one from the back ($c_{2N-1-k}$). By alternating transmission directions and executing two sub-steps per ring iteration, Zigzag Ring Attention balances the number of active lower-triangular blocks across all GPUs at every ring step, achieving:

1. 100% Compute Load Balance: Zero idle bubbles during causal decoding and training.
2. Zero Permutation Overhead: Preserves contiguous tensor layouts for standard FlashAttention-2/3 kernels.
3. Exact Causal Equivalence: Generates identical output to single-GPU causal attention.

6. Architectural Comparison: RingAttention vs. DeepSpeed-Ulysses vs. USP

Context Parallelism implementations diverge in their network communication topologies and scaling constraints.

+---------------------------------------------------------------------------------------------------+
|                       CONTEXT PARALLELISM ARCHITECTURAL COMPARISON                                |
+---------------------------------------------------------------------------------------------------+
| Dimension                | DeepSpeed-Ulysses            | RingAttention               | Unified Sequence Parallel (USP) |
+--------------------------+------------------------------+-----------------------------+---------------------------------+
| Network Topology         | All-to-All Collective        | 1D Ring P2P Asynchronous    | Hierarchical (All2All + Ring)   |
| Scaling Bound            | $N \le H$ (Attention Heads)  | Arbitrary $N$ ($N \le S$)   | $N \cdot M$ (Arbitrary Hybrid)  |
| Communication Volume     | $2 \times \frac{S \cdot d}{N}$ (All-to-All) | $2 \times S \cdot d$ (Total P2P) | Optimized 2D Grid               |
| Comm-Compute Overlap     | Difficult (Blocking All2All) | Natural (Double Buffering)  | Full Inter-Node Overlap         |
| Latency Sensitivity      | High (Requires NVLink)       | Low (InfiniBand Tolerant)   | Robust across Heterogeneous Nets|
| Causal Mask Handling     | Native (No Imbalance)        | Requires Zigzag / Striped   | Native within Node, Zigzag Ring |
+--------------------------+------------------------------+-----------------------------+---------------------------------+

DeepSpeed-Ulysses

Jacobs et al. (2023) introduced DeepSpeed-Ulysses, which uses two all-to-all collective communication operations per attention layer:

1. Input $X \in \mathbb{R}^{\frac{S}{N} \times d}$ is projected into local $Q, K, V$.
2. An All-to-All collective transposes the tensor layout from sequence-partitioned $[S/N, H, d_k]$ to head-partitioned $[S, H/N, d_k]$.
3. Standard local attention is executed over the full sequence length $S$ across a subset $H/N$ of heads.
4. A second All-to-All transposes the output back to $[S/N, H, d_k]$.

Limitation: Ulysses requires that the context parallel degree $N$ divide the number of attention heads ($N \le H$). For models with Grouped-Query Attention (GQA) where $H_{\text{KV}} = 8$, pure Ulysses cannot scale beyond 8 GPUs.

Unified Sequence Parallelism (USP)

Fang et al. (2024) unified Ulysses and RingAttention into a 2D hybrid layout:

• Intra-Node (NVLink): Runs DeepSpeed-Ulysses with All-to-All across the 8 GPUs within a single server node, maximizing high-bandwidth interconnect utilization.
• Inter-Node (InfiniBand): Runs RingAttention with asynchronous P2P ring transfers across nodes, bypassing the head-count constraint while completely hiding inter-node network latency behind computation.

7. Memory and Backward Pass Complexity

During the backward pass of RingAttention, gradients $\nabla Q$, $\nabla K$, and $\nabla V$ are computed via reverse ring rotation.

+---------------------------------------------------------------------------------------------------+
|                        RINGATTENTION BACKWARD PASS LOGIC                                          |
+---------------------------------------------------------------------------------------------------+
|  1. Forward Activation Storage:                                                                   |
|     - Retain local Q_k and final logsumexp statistics: L_k = m_k^(N) + ln(l_k^(N))                |
|     - Discard intermediate P_tilde and attention score matrices (Standard FlashAttention)         |
|                                                                                                   |
|  2. Backward Ring Recomputation:                                                                  |
|     - Rotate K and V blocks through the ring in reverse order                                     |
|     - Recompute local block attention scores S^(t) on SRAM using stored Q_k and arriving (K, V)   |
|     - Accumulate gradients dQ_k locally; accumulate dK and dV into rotating ring buffers          |
|     - Overlap dK, dV gradient buffer communication with backward GEMM execution                   |
+---------------------------------------------------------------------------------------------------+

Complete Activation Footprint Comparison

The table below summarizes per-device memory allocation across parallel training strategies for an $S = 1{,}000{,}000$ token sequence on a model with $d = 4096$, $L = 32$, and $H = 32$:

+---------------------------------------------------------------------------------------------------+
|                        PER-DEVICE MEMORY FOOTPRINT (S = 1M, N = 64 GPUs)                          |
+---------------------------------------------------------------------------------------------------+
| Parallel Strategy      | Layer Activation Memory | Communication Buffers | Peak Attention Memory  |
+------------------------+-------------------------+-----------------------+------------------------+
| Megatron TP (TP=8)     | ~65.5 GB (OOM Risk)     | ~0.5 GB (All-Reduce)  | > 80 GB (OOM)          |
| DeepSpeed-Ulysses (CP=8)| ~8.2 GB                 | ~2.1 GB (All-to-All)  | ~14.5 GB               |
| RingAttention (CP=64)  | ~1.02 GB                | ~0.25 GB (Double Buf) | ~2.8 GB                |
| Hybrid USP (U8 x R8)   | ~1.02 GB                | ~0.35 GB              | ~2.6 GB                |
+------------------------+-------------------------+-----------------------+------------------------+

8. Real-World Applications and Ecosystem Integration

RingAttention and hybrid context parallelism are implemented in major open-source training and serving stacks:

1. Large World Model (LWM): Liu et al. (2024) trained 7B and 34B multimodal models on sequence lengths of 1,000,000 tokens (representing full-length movies and podcast audio) using RingAttention on TPU v4/v5e and GPU clusters.
2. NVIDIA Megatron-LM & TransformerEngine: Context Parallelism in Megatron-LM integrates RingAttention and Ulysses with native FP8 FlashAttention-3 kernels via AttnFuncWithCPAndKVP2P.
3. vLLM & SGLang Long-Context Serving: For multi-turn agentic workflows and repository retrieval, inference engines deploy RingAttention during prefill to process 128k to 1M prompt tokens across distributed nodes without hitting single-node HBM ceilings.

Sources

• Liu, H., Zaharia, M., & Abbeel, P. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context. arXiv:2310.01889.
• Liu, H., & Abbeel, P. (2023). Blockwise Parallel Transformer for Large Context Models. Advances in Neural Information Processing Systems (NeurIPS 2023). arXiv:2305.19370.
• Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems (NeurIPS 2022). arXiv:2205.14135.
• Jacobs, S. A., Tanaka, M., Zhang, C., et al. (2023). DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. arXiv:2309.14509.
• Zhang, Z., et al. (2024). Zigzag Ring Attention: Efficient Causal Attention for Long Sequences. arXiv:2403.04746.
• Fang, J., & Zhao, H. (2024). USP: Unified Sequence Parallelism for Long-Context Transformer Model Training and Inference. arXiv:2405.07719.
• Liu, H., et al. (2024). World Model on Million-Length Video And Language With RingAttention. arXiv:2402.08268.
• Milakov, M., & Gimelshein, N. (2018). Online normalizer calculation for softmax. arXiv:1805.02867.