A neural network can score 100 percent on its training data and still fail on every new example. Many practitioners would stop training at that point and call it overfitting. In a paper posted in January 2022, a team at OpenAI reported what happened when they kept training. Long after the model had memorized its training set, it suddenly started to get new examples right as well. They called this "grokking."
This explainer covers what grokking is, what the original experiment showed, what later papers found inside the model when it happens, and what it does and does not tell you about large language models. Every number below comes from one of the papers we cite, and we name the figure or section it comes from.
The short version
Grokking is delayed generalization. A model first memorizes its training examples. Then, after much more training and with no change to the data, its accuracy on held-out examples jumps from about chance to near perfect.
- Memorize first: training accuracy reaches close to 100 percent early.
- Long plateau: accuracy on new examples stays low for a long time.
- Late jump: accuracy on new examples rises to near 100 percent, much later.
This explainer draws mainly on three papers:
- Power et al. (OpenAI, 2022) named the effect and measured it.
- Nanda et al. (ICLR 2023) took a small model apart and found the algorithm it learned.
- Liu, Michaud and Tegmark (ICLR 2023) linked grokking to the size of the model's weights and caused it on data other than arithmetic.
Two words you need: overfitting and generalization
When you train a model, you show it a training set and measure how well it does on that set. You also keep a separate validation set (or test set) that the model never trains on. Accuracy on that second set tells you whether the model learned the general rule or only memorized the answers.
- Overfitting means the model does well on the training set but badly on new data. It memorized.
- Generalization means the model does well on data it never saw. It learned the rule.
A common practice is to stop once the training score is high and the validation score stops improving, on the view that more training will not help. Grokking shows that this can be wrong, at least on the tasks these papers tested.
The original experiment
The paper is "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets" by Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin (OpenAI) and Vedant Misra (Google, who the paper notes was at OpenAI at the time of the work). It was posted to arXiv in January 2022 as 2201.02177.
The researchers did not use text from the internet. They used small, made-up math tables. Each example is an equation such as "x ◦ y = z", where x and y are numbers from 0 to 96 and ◦ is an operation like addition or division modulo 97. Modulo 97 means you keep only the remainder after dividing by 97, so the numbers wrap around like a clock with 97 positions. Each number is its own token, so the model sees symbols, not digits.
For each run, the team picked a fraction of all possible equations at random for training. The rest became the validation set. The model had to fill in the missing cells of the table.
The model was small. The appendix describes a standard decoder-only transformer with 2 layers, width 128 and 4 attention heads, about 400,000 non-embedding parameters. Most runs used the AdamW optimizer with weight decay set to 1.
What they saw
Figure 1 of the paper shows the headline example: division modulo 97, with 50 percent of the equations in the training set.
- Training accuracy was close to perfect in fewer than 1,000 (10^3) optimization steps.
- The team saw very little sign of generalization until about 100,000 (10^5) steps.
- Validation accuracy reached the training level at close to 1,000,000 (10^6) steps.
In the text, the authors say validation accuracy started to rise above chance only after 1,000 times more steps than training accuracy needed to get close to its best. For this run the team used the Adam optimizer with no weight decay and a longer budget of 10^6 steps, to show how late generalization can begin (Appendix A.1.2).
The paper also reports three broader findings:
- Less data, much longer wait. In Figure 1 (center), for the product of elements in the group S5 (a group of ways to reorder 5 items), the steps needed to reach 99 percent validation accuracy grew quickly as the training fraction fell. Near 25 to 30 percent of the data, removing 1 percent of the training data raised the median time to generalize by 40 to 50 percent. Steps to reach 99 percent training accuracy stayed in the range of 1,000 to 10,000.
- Weight decay helped most. Section 3.3 compares several training changes, such as dropout, noise and different batch sizes. Adding weight decay had "a very large effect on data efficiency," more than halving the samples needed compared to most other interventions. Weight decay is a setting that pulls weights toward zero during training.
- Some tables never clicked. For some operations, such as x^3 + xy^2 + y modulo 97, the model did not generalize within the step budget at any training fraction up to 95 percent. It only memorized.
The authors also noticed that symmetric operations, where swapping x and y gives the same answer (for example x + y), needed less data than closely related operations where order matters (for example x - y).
What happens inside the model
Power et al. described what happens. They did not explain why. A year later, Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith and Jacob Steinhardt published "Progress measures for grokking via mechanistic interpretability" (arXiv 2301.05217, ICLR 2023). Mechanistic interpretability means reverse engineering a trained network into parts you can read, like reading compiled code.
Their main setup, from Section 3:
- Task: addition modulo 113.
- Model: a one-layer transformer with 4 attention heads and a 512-unit MLP layer.
- Data: 30 percent of all 113 × 113 number pairs for training. The rest for testing.
- Training: full-batch AdamW with weight decay 1, for 40,000 epochs.
The model memorized the training set first. After around 10,000 epochs, test accuracy rose to near 100 percent.
The algorithm it found
The team reverse engineered the trained model and found that it does addition with waves. In plain words: the model turns each input number into a position on a circle, using sine and cosine at a few fixed frequencies. Adding two numbers then becomes rotating around the circle. The model then reads off which answer matches the final angle. The paper describes it as using "discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle."
They checked this in several ways. In the introduction, they report that removing the few key frequencies the model uses dropped performance to chance, while removing the other 95 percent of frequencies slightly improved it.
The three phases
The interesting part for grokking is timing. Using measures built from that algorithm, the authors split training into three phases (Section 5.2):
- Memorization (epochs 0 to 1,400). Training loss falls. Test loss stays high. The model stores the answers.
- Circuit formation (epochs 1,400 to 9,400). The wave-based circuit grows inside the model. The total size of the weights falls. Test accuracy still looks flat.
- Cleanup (epochs 9,400 to 14,000). Weight decay removes the memorized parts, because the general circuit does the job with smaller weights. Test loss suddenly drops.
So the "sudden" jump is not sudden inside the model. The general solution was being built for thousands of epochs while the test score stayed flat. The jump in the score came when the memorized solution was cleared away. The abstract says grokking "rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components."
Section 5.3 adds that in their tests the networks did not grok on this task without weight decay or some other form of regularization, and smaller weight decay made grokking take much longer. With enough training data, the gap disappeared: train and test loss fell together.
Why weight size matters
A third paper gives a simpler picture. Ziming Liu, Eric Michaud and Max Tegmark's "Omnigrok: Grokking Beyond Algorithmic Data" (arXiv 2210.01117) looks at how training and test loss change with the total size (norm) of a model's weights.
Their claim, from the abstract: plotted against weight norm, training loss usually looks like an "L" and test loss looks like a "U". They call this the "LU mechanism."
- With large weights, the model can still fit the training data well, so training loss is low.
- But test loss is high there. Good generalization sits in a narrower band of weight sizes.
- If a model starts with large weights, it fits the training data quickly, then weight decay slowly shrinks the weights toward the band where test loss is low. That slow drift is the long wait.
Using this picture, the authors report that they could cause grokking on images (MNIST), language (IMDb reviews) and molecules (QM9) by starting with larger weights. They also report they could remove grokking from algorithmic tasks. They write that standard initialization schemes usually do not start weights that large, which is one reason grokking is not commonly seen. They also say grokking is more dramatic on the algorithmic tasks than on the others.
An earlier paper by some of the same authors, "Towards Understanding Grokking" (arXiv 2205.10343), maps training outcomes into four phases (comprehension, grokking, memorization and confusion) and reports that good representations form only in a middle zone between memorization and confusion.
What grokking tells you about large language models
Be careful here. All of the results above come from small models on small, clean tasks. None of these papers trains a chatbot-sized model and shows it grokking. So what can you take from them?
- A flat score can hide progress. Nanda et al. found a general circuit forming while the test score did not move. Their stated goal was progress measures, metrics that change smoothly before a sudden jump. That idea could help predict when a large model will gain a new skill, but the authors say their measures are specific to this one small task and cannot yet predict when a jump will happen. The paper frames grokking as a case study for emergent behavior in general.
- Regularization changes what a model learns. In Nanda et al.'s runs, the model did not move from the memorized solution to the general one without weight decay or another regularizer. In Power et al., weight decay was the most effective of the training changes they tried, though their Figure 1 run grokked without it.
- Stopping early has a cost on some tasks. If a run had stopped when validation accuracy stalled, it would never have reached the general solution. On these tasks, the general solution came only with much more training.
- Data amount changes the picture. Both Power et al. and Nanda et al. report that with more training data, the gap between memorizing and generalizing shrinks or goes away.
What the papers do not show: that large language models suddenly "understand" things because of grokking, or that any behavior in a production model comes from it. If you read a claim like that, check whether the source tested a large model or is extending results from toy tasks.
Common questions
Is grokking the same as overfitting? No. Overfitting is the first stage. Grokking is what can happen after it: generalization that arrives long after the model has memorized.
Is it the same as double descent? Not quite. Power et al. note that the validation loss in their division run shows a double descent, where loss falls, rises, then falls again. But the authors say grokking might be distinct from the double descent described in earlier work: their second drop in loss comes far past the point where training loss is already very small, and accuracy does not dip and recover. Grokking is about generalization arriving late in training time, on a fixed dataset.
Does it need a transformer? The first two papers used transformers. Omnigrok reports grokking with other networks too, including a simple multilayer perceptron (a basic feed-forward network) on MNIST, under its large-initialization setup.
Can I see it myself? Power et al. note that these experiments can be quickly reproduced on a single GPU. The setups above are small: a table of numbers, a tiny model and weight decay.
Further reading on llms.blog
For background on the model type used in these papers, see our plain-English guide to transformers. For how scale affects model quality, see AI scaling laws explained. For another case of a model picking up a skill without new weights, see in-context learning explained.
For a more technical treatment of the same effect, with the math behind weight decay and circuit efficiency, see our deep dive on grokking and delayed generalization.
Sources
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, Power, Burda, Edwards, Babuschkin and Misra, 2022
- Progress measures for grokking via mechanistic interpretability, Nanda, Chan, Lieberum, Smith and Steinhardt, ICLR 2023
- Omnigrok: Grokking Beyond Algorithmic Data, Liu, Michaud and Tegmark, ICLR 2023
- Towards Understanding Grokking: An Effective Theory of Representation Learning, Liu, Kitouni, Nolte, Michaud, Tegmark and Williams, 2022
