Looped transformers run the same block of layers several times per token. That buys extra depth without adding trainable weights. The catch is training cost: every extra loop adds a forward pass and, if you backpropagate through it, a backward pass too. So several looped language models cut the backward pass short and only train the last few loops.
A new paper from MMLab at the University of Hong Kong and Yale, Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation (arXiv 2610.26030; the PDF is dated October 15, 2026, v1 was posted October 19 and v2 October 20), tests that shortcut with nearly 600 training runs. Its answer: cutting the backward pass only pays if someone has already fixed how many loops the model runs. When you can choose both numbers, the fitted law says to use fewer loops and train all of them.
What a looped transformer is
The paper uses the prelude, recurrent block and coda layout from Huginn (Geiping et al., 2025), a recurrent-depth model built for latent reasoning. A prelude turns tokens into hidden states. A shared recurrent block refines those states T times. A coda turns the result into next-token predictions.
Training uses backpropagation through time (BPTT), the same method used for recurrent networks. Full BPTT sends gradients through all T loops. Truncated BPTT runs the early loops with no gradient tracking and only trains the last B loops. The paper writes each setting as (T, B). So (4, 2) means four forward loops, with gradients through the last two.
Truncation saves backward compute and activation memory. It also gives biased gradients. The paper notes that several looped models use it, and that an earlier study, Iso-Depth (Schwethelm et al., 2026), found that truncating to half the loops lowered loss at a fixed forward loop count. That study and Parcae (Prairie et al., 2026) fit laws for set recipes. Parcae varied backward loops in small-model tests but fixed them at half the forward loops in its scaling study. Neither law treated the backward loop count as its own variable.
The setup
The authors follow the Iso-Depth training recipe. They train five model sizes from 58M to 919M non-embedding parameters on the 100B-token sample of FineWeb-Edu, with up to 16 forward loops and token budgets that span more than three orders of magnitude in tokens per parameter. They use MuonH for the transformer block matrices and AdamW for embeddings, output and norm layers. The metric is validation cross-entropy in nats per token on held-out shards.
They fit one scaling law for all settings. It expresses each loop setting as an "effective parameter count": the size of a non-looped model that reaches the same loss on the same tokens. Fully trained loops raise the effective count, with diminishing returns. Truncation discounts that gain. The fitted exponents are about 0.237 for forward loops and 1.022 for backward loops.
Finding 1: truncation helps only when T is fixed
If the forward loop count is set in advance, the law agrees with earlier work. At T = 8, it picks B = 2 and predicts a 34% compute saving at equal loss. Separately fitted laws give 5 to 11% less compute at T = 4 and 33 to 35% less at T = 8, where the saved compute goes partly into a larger model.
But T is usually a design choice. Under the paper's FLOP accounting, a forward pass through the block costs one unit and a backward pass costs two, so (4, 4) and (8, 2) cost the same. With both counts free and treated as continuous, the law's optimum is full BPTT at every compute level. With whole-number loop counts, the paper notes that a setting such as (4, 3) can lead (2, 2) and (4, 4) by at most 0.002 nats over a narrow compute range. Dropping forward loops frees compute to train every remaining loop, and that adds more effective parameters than the extra untrained loops did.
The authors checked this with matched runs at 17.2 billion tokens (Table 6). In each row, (4, 4) and (8, 2) share model size, tokens and training compute. Full BPTT had lower loss at all five sizes:
- 60M: 2.5211 vs 2.4933 nats, a 0.0278 drop
- 120M: 2.3843 vs 2.3696, a 0.0147 drop
- 240M: 2.2778 vs 2.2642, a 0.0136 drop
- 480M: 2.1845 vs 2.1738, a 0.0107 drop
- 1B: 2.1144 vs 2.1057, a 0.0087 drop
The gap gets smaller as the model gets larger, but it stays in favor of full BPTT in every row.
Finding 2: at a fixed size, put most new compute into data
Suppose memory limits the model size. More compute can go to more loops per token or to more tokens. Under the law with full BPTT, the best loop count grows about as C^0.24 and the token count as C^0.76 in the large-compute limit. So ten times more compute raises the best loop count about 1.7 times and the token count about 5.8 times. The paper states these trade-offs as powers of compute, the same way the Chinchilla scaling laws do. Most extra compute should go to data, and loops should go up slowly.
Finding 3: with free size, no looping wins, but two loops come close
When model size is also free, the law favors a plain non-looped model at every compute level. The paper says this is consistent with Iso-Depth. Among looped settings, (2, 2) ranks first, because two shared loops give nearly the effective size of two untied copies of the block.
In runs at three budgets, with three seeds each, the compute-optimal (2, 2) model stayed within 0.008 nats of the compute-optimal non-looped model, with 27 to 32% fewer non-embedding parameters. At 2.1 x 10^19 FLOPs, the authors trained the (2, 2) model on 8% more tokens, and so with 8% more compute. Its loss (2.36 nats) then fell below the non-looped model (2.37 nats), with 344M versus 471M non-embedding parameters, which is 27% fewer.
Why it matters
For teams that train looped or recurrent-depth models, the results point to a simple rule. Do not pick a large loop count and then truncate to afford it. Pick a smaller loop count you can train fully. And if parameter count matters, for example on a device with limited memory, two fully trained loops get close to a larger standard model for a small amount of extra training compute.
Limits
The authors list them. All models are dense, so the results may not carry over to looped mixture-of-experts models. They measure validation loss only, not downstream tasks or reasoning. Runs stop at 16 loops, so behavior beyond that is extrapolation. They also fix the share of parameters in the prelude and coda versus the recurrent block for each size. Code is on GitHub.
