Subscribe
AI
6 min read

What Is Model Collapse? How AI Models Degrade by Training on Their Own Output

Every generative model is a copy of the data it was trained on. Model collapse is what happens when the data being copied is other models. First formally described by researchers at Oxford, the University of Toronto, and their collaborators, the effect is one of the few well-documented failure modes that becomes more probable, not less, as AI-generated text floods the web. The basic mechanism The canonical paper, published in Nature in July 2024, defines the phenomenon precisely: "indiscrimin

What Is Model Collapse? How AI Models Degrade by Training on Their Own Output

Every generative model is a copy of the data it was trained on. Model collapse is what happens when the data being copied is other models. First formally described by researchers at Oxford, the University of Toronto, and their collaborators, the effect is one of the few well-documented failure modes that becomes more probable, not less, as AI-generated text floods the web.

The basic mechanism

The canonical paper, published in Nature in July 2024, defines the phenomenon precisely: "indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear."

The tails are the point. A model trained on human text learns both the common patterns and the rare ones: unusual vocabulary, fringe topics, minority writing styles, edge cases. When it generates new text, it reproduces the common patterns faithfully and samples the rare patterns imperfectly, or not at all. A second model trained on that output inherits the shrunken version of the distribution. Repeat the loop and the rare stuff is gone. What is left is a narrower, smoother, more self-confident model that is measurably worse at representing the world it was trained to imitate.

The authors attribute collapse to three compounding sources of error: statistical approximation error (finite samples miss rare events), functional approximation error (no model captures the true distribution exactly), and learning error (optimization never converges perfectly). Each generation adds noise. None of it gets subtracted.

The math does not require neural networks. In the simplest possible case, a one-dimensional Gaussian re-estimated from samples of itself: each generation's mean and variance pick up sampling noise, the parameters perform a random walk, and the distance from the original distribution grows with the number of generations. Holding the per-generation dataset size constant, the drift is linear. To keep the error bounded, each generation would need to sample strictly more data than the last.

What it looks like in practice

Model output feeding back into training as a self-consuming loop

The earliest demonstrations were vivid. In the 2023 preprint that became the Nature paper, a language model retrained on its own output for successive generations eventually answered a question about medieval church architecture with "the world's largest populations of black @-@ tailed jackrabbits, white @-@ tailed jackrabbits, blue @-@ tailed jackrabbits..." The team called the effect the curse of recursion and extended the name to the gibberish itself.

Successive generations narrowing a distribution until its tails disappear

Quantitative results are less dramatic and more important:

  • A study presented at FAccT 2024 separated the two phases the Nature paper describes. Early model collapse quietly deletes performance on minority data while average benchmark scores hold steady or even improve. Late collapse is the visible failure: confusion, repetition, lost variance. The uncomfortable finding is that early collapse is nearly invisible in aggregate metrics, which means a model can be degrading on rare inputs long before anyone's eval catches it.
  • Research on linguistic diversity found that iteratively training on synthetic text produces consistent declines in lexical, syntactic, and semantic diversity, with the steepest losses on tasks that need creativity.
  • A theoretical treatment of recursive training in high-dimensional regression (NeurIPS 2024) derived closed-form error growth: test error rises linearly with the number of model generations, and in the over-parameterized regime the collapse can be catastrophic, growing exponentially even with noiseless labels.
  • A follow-up on scaling laws introduced "strong model collapse": even a fixed, unchanging fraction of synthetic data in the corpus changes the scaling law itself, buying less capability per token than natural data does. The model does not have to degenerate into gibberish for the synthetic fraction to cost it.

The honest counterargument

The 2024 debate produced a genuine correction to the panic. The Nature experiments assumed each generation retrained on fresh synthetic data that replaced the previous corpus. A group at Princeton and Yale showed that this replacement assumption does not match how training corpora actually work: web crawls accumulate. Their conclusion, from both theory and experiments: "replacing the original real data by each generation's synthetic data does indeed tend towards model collapse," but accumulating synthetic generations alongside the original real data avoids it. Rare events keep getting re-seen from human sources, and the distribution tails survive.

The practical disagreement is smaller than the headlines suggested. Collapse is a property of the data pipeline, not a prophecy. A pipeline that deletes last year's corpus and retrains on this year's AI output is on the collapse path. A pipeline that adds human text to a growing pile is not, at least not for this reason.

There is also a separate literature reminding us that not every sameness problem is model collapse. RLHF and preference fine-tuning narrow output diversity too, without any synthetic data involved; that is a generalization-diversity trade-off in the alignment recipe, mechanistically distinct from recursive training. Attributing every boring model output to "model collapse" is usually wrong.

Why labs still use synthetic data

If the risk is real, why is synthetic data everywhere? Because the alternative is running out of human text, and because synthetic data, when generated by a better model and validated, behaves well. Work on large-scale synthetic pre-training corpora (COLM 2025) found that carefully generated synthetic data follows predictable scaling laws like natural data, with gains plateauing as volume grows. This is distillation economics: synthetic data carries the knowledge already in the teacher model at a discount, but it cannot add knowledge the teacher does not have. The collapse literature explains why that ceiling exists and why fresh human data is the only thing that lifts it.

The Nature authors put the economic consequence plainly: the value of data collected about genuine human interaction with systems becomes more valuable as AI-generated content floods the crawled web. Clean-room human corpora, licensed publications, and first-party interaction logs are the non-renewable resource of this era of training.

What mitigations actually work

Everything published on this points the same direction, because the mechanism is statistical, not mystical:

  1. Keep accumulating real data. Do not let synthetic generations replace the human corpus; treat them as additive at most (Gerstgrasser et al., 2024).
  2. Filter aggressively. In the self-consuming loop framework known as MAD, generations only improved on the original when synthetic samples were first screened by a quality classifier; unfiltered, the loop converged to a degenerate fixed point.
  3. Detect provenance. LLM watermarking exists partly so pipelines can identify and exclude model-generated text before it re-enters training data. It is an arms race with a lag; treat it as one filter among many.
  4. Buy or license human text. The scarcity is now priced: data curation contracts increasingly distinguish human-origin content from AI-generated web litter.
  5. Watch distribution tails, not averages. Since early collapse hides inside aggregate metrics (Wyllie et al., 2024), evaluation suites need per-slice and rare-case reporting to catch it.

FAQ

Is model collapse the same as a model getting old? No. A frozen model does not degrade over time. Collapse refers to retraining successive generations on model output.

Does one contaminated dataset ruin everything? No. With accumulating real data the effect is bounded (Gerstgrasser et al., 2024), though a fixed synthetic fraction still bends scaling laws (Dohmatob et al., 2024).

Is synthetic data forbidden in training? No. Distillation-quality synthetic data, generated from stronger validated models and mixed with human data, is standard and effective. The failure mode is indiscriminate use of unvalidated model output in recursive retraining.

Sources

AILLMEducationExplainers