Google has added long-context training to MaxText, Google Cloud's open-source JAX training framework, so one very long training sequence can be split across many Cloud TPU chips. The feature is called context parallelism. It is available from MaxText v0.2.4, and Google says it took Llama 3.1 8B training to 14 million tokens of context in its own benchmarks.
This is a training software feature, not a new model or a consumer product. It matters to teams that pretrain or post-train models on Cloud TPU and want those models to handle very long inputs.
Why is long-context training hard?
Google says context windows have grown in three years from 128K to 200K tokens to 1M to 2M tokens, and that models trained toward 10M tokens are arriving. In the post's words: "Training at those lengths, not just serving them, is where the system limits show up."
The announcement names two bottlenecks:
- Attention compute. At long context, attention math grows quadratically with sequence length, and attention dominates the training step.
- Memory and communication. Activations, including the keys and values (KV) for every token, grow with sequence length. If every chip keeps a full copy, as a naive all-gather does, the chips run out of high-bandwidth memory (HBM) and the job fails.
What is context parallelism?
Context parallelism splits a long sequence across chips, so no chip holds the activations of the whole sequence. Attention is the hard part, because every token needs the keys and values of every earlier token. Each strategy brings those keys and values back together in a different way, and each trades memory, communication and model shape against the others.
Google says context parallelism can be combined with data and tensor parallelism, and that "the smallest context-parallel size that fits your sequence in HBM is usually the right one."
For how these methods compare outside MaxText, see our guide to context parallelism in production, which covers DeepSpeed Ulysses, ring attention and USP.
The four strategies in MaxText
- All-gather. Each chip keeps its own chunk of the sequence and, at every layer, gathers the keys and values from every other chip. Google calls it the simplest option. Because later tokens attend to longer prefixes, it needs load balancing across chips.
- All-to-All (DeepSpeed Ulysses). This splits attention heads across chips instead of tokens. It needs no token load balancing. But you cannot split across more chips than the model has heads, and under grouped-query attention the cap is the KV-head count: eight for Llama 3 8B, per Google. Its all-to-all step does not overlap with compute.
- P2P (Ring Attention). KV chunks travel around a ring of chips one hop at a time. Each chip works on the chunk it holds while the next one is in flight, so communication overlaps compute. Smaller chunks can lower kernel efficiency, and it needs explicit load balancing.
- A2A + P2P hybrid (Unified Sequence Parallelism, USP). All-to-All runs along one axis of the chip mesh and a ring along the other. Each ring step carries a bigger chunk, and the ring is shorter.
All four run on Splash Attention, a block-sparse flash attention kernel from Tokamax, Google's open-source library of TPU kernels. Combined with the 3D inter-chip interconnect on TPUs such as TPU 5p and TPU 7 (Ironwood), Google says this "lets one sequence span an entire slice while communication overlaps compute and memory per chip stays flat."

What do Google's benchmarks show?
All figures below are Google's own, from its blog post:
- Llama 3.1 8B reached 70% Model FLOPs Utilization (MFU) on TPU 5p for sequence lengths up to 4 million tokens. MFU is the share of the chips' peak math throughput that a training run actually uses.
- Across TPU 5p and TPU 7, context was extended to 14 million tokens, with TPU 7 "consistently maintaining 50% MFU across all sequence scales below 6 million tokens."
- On a fixed 64-chip TPU 5p slice, Google says that "in ideal settings" P2P can outperform All-gather at almost every context length from 64K to 14M tokens.
- With twice the HBM per chip, P2P training of 14M tokens on 64 TPU 7 chips reached 48.7% MFU, compared to 37.1% on TPU 5p.
- Google "observed that ring attention performs better than other context parallelism strategies," such as Ulysses and USP, on TPU 7 chips.
These are vendor benchmarks on one model family. The post includes no independent results, and real jobs will vary with model shape, batch size and slice size.

Which strategy should teams choose?
Google's guidance depends mostly on whether the gathered KV fits in HBM and on how many KV heads the model has:
- Gathered KV fits in HBM: use All-gather, with load balancing on. Google says it "was the fastest or near fastest in our runs."
- A few chips, up to the KV-head count: use All-to-All alone. It does not support multi-query attention.
- Gathered KV no longer fits: use the ring family. Google recommends USP "whenever the KV heads allow an A2A factor of 2 or more."
- DeepSeek-style multi-head latent attention (MLA) models, or when the head count does not divide: use P2P alone.
For low memory, Google's rule is "Offload before you recompute." MaxText can move saved activations, optimizer states and parameters to host memory. Google's suggested order is attention outputs first, then optimizer states, layer inputs, parameters, and recomputation (remat) as a last resort.
How do you turn it on?
Google says the feature is available in MaxText v0.2.4 on PyPI, in the matching Docker images, and on GitHub main, with sample recipes for TPU 7 and TPU 5p. The MaxText release notes list the Ulysses and USP strategies under v0.2.4, and newer releases have followed. You pick a strategy with the context_parallel_strategy flag (the post lists ring, ulysses and usp; the MaxText sharding guide also lists all_gather), set the context-parallel size, "and keep the rest of your run the same," according to the post.
What comes next?
Google lists two planned directions. The first is broader model coverage: models that mix sliding-window and global layers, such as Gemma, plus linear attention methods such as Gated Delta Net (GDN) and Kimi Delta Attention (KDA). The second is performance: more tuning of the Tokamax kernels and the P2P backward pass "to narrow the remaining gap to all-gather at short context," and recipes by model and context length for TPU 5p and 7. MaxText's v0.2.6 release notes already list a sliding-window context-parallel mode for Gemma 2, 3 and 4.
Why it matters
Google's post calls long-context training a foundation of frontier AI models, and argues that training at those lengths, not only serving them, is where the hard limits are. MaxText's update puts four known context-parallel methods behind one flag on Cloud TPU, with Google's own guidance on when to use each.
It also fits a pattern of Google opening more of its TPU training stack to Cloud customers. Earlier this week Google made its Pathways training system available on Google Cloud, and it recently detailed new TPU 8tx and 8i Max systems. For teams already on TPUs, the main open question is how close their own models get to Google's benchmark numbers.
