Subscribe
AI
6 min read

What Is Overfitting? How Language Models Memorize Their Training Data

In classical machine learning, overfitting is the failure mode students meet first. A model trained too hard on too little data stops learning the pattern and starts memorizing the examples, noise included. The symptom has a fixed shape: error on the training set keeps falling while error on held-out data rises. The model flatters itself and surprises no one else. For a long time, the consensus was that large language models had outgrown this problem. The GPT-3 paper noted that the model was tr

What Is Overfitting? How Language Models Memorize Their Training Data
Cover

In classical machine learning, overfitting is the failure mode students meet first. A model trained too hard on too little data stops learning the pattern and starts memorizing the examples, noise included. The symptom has a fixed shape: error on the training set keeps falling while error on held-out data rises. The model flatters itself and surprises no one else.

For a long time, the consensus was that large language models had outgrown this problem. The GPT-3 paper noted that the model was trained on less than one epoch of its data and showed no meaningful gap between train and test loss, something earlier, smaller models never achieved. If a model sees each example at most once and still generalizes, the logic went, it cannot be overfitting.

Overfitting did not disappear. It just changed shape. Understanding how is one of the clearest windows into how these models actually learn.

Memorization without the train-test gap

In 2021, Nicholas Carlini and colleagues at Google, OpenAI, Stanford and elsewhere published a direct rebuttal to the no-overfitting consensus. Working with only black-box access to GPT-2, they generated candidate strings, ranked them by how much more likely GPT-2 found them than a reference model did, and checked the winners against the training set. Of the 1,800 candidates they manually inspected, 604 were verbatim training data. Out of one model they extracted a named individual's address, email and phone numbers, IRC conversations, executable code, and 128-bit UUIDs that appeared in exactly one training document.

The finding they emphasized is in the paper title's companion claim: memorization does not require overfitting. These models had small train-test gaps. They memorized anyway. Two variables predicted what got remembered: model size, which scales memorization roughly log-linearly, and repetition, which matters more than anything else. A string inserted 33 times into the training set could be reproduced word for word by the largest model tested.

This reframes the concept. Textbook overfitting is one mechanism that produces memorization. Language models reach the same destination by a different road: at scale, with one pass over trillions of tokens, enough statistical weight lands on repeated sequences that they get written into the parameters verbatim.

Punched data cards fused permanently into a circuit lattice while others pass through as outlines

The data is the culprit

Why would a model trained on the entire web reproduce one UUID out of a trillion tokens? Because the web repeats itself. Scraped corpora are full of duplicated documents, and repeated content is exactly what deep networks memorize.

Two 2022 studies made this precise. Kandpal, Wallace and Raffel found that the rate at which a model regenerates a training sequence grows superlinearly with how many times that sequence appears in the training set. Sequences appearing once were essentially never regurgitated; duplication did the damage. They also showed that deduplicating the training data cut the emission of training sequences by roughly 20 times. Lee and colleagues at Google DeepMind and Stanford, in a companion study, found that over 1% of unprompted model output was copied verbatim from standard web-scraped datasets (C4 alone contained a single 61-word sentence repeated more than 60,000 times), and that deduplication reduced memorized emissions tenfold while also shrinking the train-test overlap that quietly corrupts evaluation.

The practical lesson became a standard part of pre-training pipelines: deduplication is not just a quality or compute-saving measure. It is the primary defense against a model handing back its training data.

Overtrained on purpose

A second twist confuses the vocabulary. In classical practice, more training on the same data is a mistake to avoid. Frontier labs now do a version of it deliberately.

The Chinchilla scaling analysis from DeepMind in 2022 computed the compute-optimal trade-off between parameters and tokens, and found that the big models of the day, GPT-3 included, had too many parameters and too little data. Labs responded by training smaller models on more data. Then most went considerably further than Chinchilla optimum itself. The Llama 3 report describes models trained on about 15 trillion tokens; for the 8B model in the family that works out to roughly 1,875 tokens per parameter, orders of magnitude past the Chinchilla point. This is not negligence. Sardana and colleagues showed why the deviation makes sense: Chinchilla optimizes training compute alone, but a model deployed for inference pays per token served. Training a smaller model longer, into what they measured as an overtraining regime of up to 10,000 tokens per parameter, still improved quality and cut total serving cost. The field adopted a term for this that would be an insult in a machine learning course: overtrained, as a design goal.

There is a limit, though, and it is the classical one. When labs run out of fresh data and start repeating epochs, the gains stop. Muennighoff and colleagues' scaling study for data-constrained training found that performance degrades once a dataset is repeated beyond roughly four epochs, with the optimal number of repetitions shrinking as datasets grow. Overtraining on new tokens generalizes. Repeating the same tokens memorizes, and the industry knows which one it prefers.

Two unlabelled curves that start together and diverge: continued training on fresh data versus repeated data

Where overfitting still bites

Three places matter in day-to-day LLM practice.

Fine-tuning. Adaptation jobs use thousands of examples, not trillions of tokens, and train for many epochs. This is classical small-data territory, and classical remedies apply: watch held-out loss, stop early, use low-rank methods that constrain how much the weights can move.

Benchmarks. A model that improves against a fixed public test set may partly be fitting the test set, through training exposure or tuning pressure. This contamination problem is large enough that benchmark suites like LiveBench now refresh their questions monthly specifically to stay ahead of it.

Alignment. Post-training does not remove memorization. Carlini and colleagues' follow-up work on production models extracted gigabytes of training data from open, semi-open and closed models including ChatGPT, using a divergence attack that pushed the model out of its chat persona. In 2024, Rando and Tramèr showed that the cheap way in was fine-tuning: for under 3 dollars of API fees, they tuned open models that regurgitated arbitrary training text the base model refused to emit. Safety training suppresses output; it does not erase the stored text.

The useful mental model

Overfitting in the textbook sense, a widening train-test gap, is mostly absent from frontier pre-training. The underlying failure it names, a model buying memorized specifics instead of general structure, is alive and well. It shows up as regurgitated web pages, leaked personal data, benchmark scores that do not transfer, and fine-tuned chatbots that repeat whatever the base model quietly stored.

The research agenda reflects that shift. The defenses are data-side (deduplication, licensing clean sets like the Pile alternatives the research community has built), optimization-side (weight decay, which in toy settings is precisely what pushes models from memorization to generalization, even producing the delayed-generalization behavior known as grokking), and privacy-side (training with differential privacy guarantees, which the Carlini paper showed can prevent extraction outright, at a cost in utility that improved models have been paying down since).

A language model reading the internet is a model with a powerful incentive to remember the internet. The engineering question was never whether it overfits in the classroom sense. It is which parts of the training data the model is allowed to keep.

Sources

AILLMexplainer