Why did AI labs spend years building ever larger models, and why did a 2022 paper argue they had the balance wrong? Much of the answer sits in a small set of research papers on "scaling laws." These are measured rules of thumb that predict how much better a language model gets when you give it more parameters, more training data, or more computing power.
This explainer covers what scaling laws are, what the two best-known papers found, why they disagreed, and how a newer line of work applies the same thinking to the time a model spends answering a question. Every claim below comes from the papers linked in the text.
What a scaling law is
A scaling law is a pattern you find by experiment, not a law of physics. Researchers train many models of different sizes on different amounts of data, measure how good each one is, and look for a simple curve that fits all the results. If the curve holds, you can use it to predict how a much bigger model will do before you pay to train it.
For language models, "how good" is usually measured with the test loss. In plain terms, the loss tells you how surprised the model is by the real next word in text it has not seen. Lower loss means better predictions.
Three ingredients go into the curve:
- Model size: the number of parameters, the adjustable numbers inside the network.
- Dataset size: how much text the model trains on, usually counted in tokens (word pieces; see our explainer on tokens).
- Compute: the total amount of calculation spent on training. More parameters and more tokens both cost more compute.
The question every lab faces is practical. If you have a fixed compute budget, how should you split it between a bigger model and more data?
The 2020 paper: bigger models, modest data
In January 2020 a team with first author Jared Kaplan published Scaling Laws for Neural Language Models. Its main finding is in the first lines of the abstract: the loss "scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude."
A power law is a curve where each time you multiply the input by some factor, the output improves by a fixed proportion. On a chart with logarithmic axes it looks like a straight line. That straight line is what made the result so useful: you can extend it to the right and read off a prediction.
The paper also reported that other design details, such as the network's width or depth, "have minimal effects within a wide range." In other words, for loss, the raw scale mattered much more than the exact shape of the network.
The finding with the biggest practical effect was about efficiency. According to the abstract, larger models are "significantly more sample-efficient," which means they learn more from each token they see. From this the authors concluded that compute-efficient training "involves training very large models on a relatively modest amount of data and stopping significantly before convergence." Put simply: build the biggest model you can afford and do not worry about training it to the end.
The 2022 Chinchilla paper: scale data too
Two years later a team with first author Jordan Hoffmann published Training Compute-Optimal Large Language Models, now usually called the Chinchilla paper. It asked the same budget question and reached a different answer.
The team trained "over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens." From those runs they concluded that model size and training tokens "should be scaled equally: for every doubling of model size the number of training tokens should also be doubled."
That led to a blunt claim in the abstract: "current large language models are significantly undertrained." The paper links this to the field's focus on making models bigger while keeping the amount of training data about the same. For the math behind the result, see our deep dive on Chinchilla scaling laws.
The test: Chinchilla against Gopher
The authors did not stop at a prediction. They trained a model sized by their own rule, called Chinchilla, and gave it the same compute budget as Gopher, an earlier 280 billion parameter model. Chinchilla has 70 billion parameters, a quarter of Gopher's size, and trained on 4 times more data.
The paper reports that Chinchilla "uniformly and significantly outperforms" four larger models on a wide range of downstream tasks:
- Gopher (280B parameters)
- GPT-3 (175B)
- Jurassic-1 (178B)
- Megatron-Turing NLG (530B)
On the MMLU benchmark, a broad multiple-choice test of knowledge across many subjects, Chinchilla reached an average accuracy of 67.5 percent, which the authors describe as more than a 7 percent improvement over Gopher.
Why the two papers disagree
Both papers measured real trends, and both gave advice about splitting a fixed compute budget. The difference is in the advice:
- The 2020 paper: put most of the extra budget into a larger model, and train on a relatively modest amount of data.
- The 2022 paper: grow the model and the data together, at the same rate.
In our reading, the Chinchilla result was the stronger practical test, because the authors trained the model their rule predicted and compared it head to head against larger models built on the same budget. The Chinchilla paper also gives a reason for the gap: the 2020 authors "use a fixed number of training tokens and learning rate schedule for all models," which, the 2022 authors argue, pushed the older analysis toward growing model size faster than data. A curve fit on one set of training choices can point in a different direction when the setup changes.
Why it matters outside research labs
Scaling laws can sound like a topic only for people who train models. They affect anyone who uses one, for three reasons.
Smaller models can be just as good. If a 70 billion parameter model can beat a 280 billion parameter one, you do not always need the biggest model. The Chinchilla abstract notes that a smaller model "uses substantially less compute for fine-tuning and inference," which "greatly" helps downstream use. Inference is the work a model does each time it answers you, so a smaller model is cheaper and faster to run every day, not only to train once.
Data becomes the bottleneck. If tokens should grow as fast as parameters, then a lab that wants a much stronger model needs much more text. The Chinchilla authors wrote that "larger, high quality datasets will play a key role in any further scaling of language models."
Labs can plan before they spend. A curve that predicts loss lets a team test ideas on small, cheap models and estimate what the large version will do.
The newer frontier: scaling the answer, not only the training
The first scaling laws were all about training. A newer line of research asks a different question: what happens if you let a model spend more computation while it answers a question?
The 2024 paper Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters studies this directly. "Test-time compute" means the work a model does at the moment you ask it something, for example trying several answers and checking them, or revising its own answer step by step. Our technical explainer on test-time compute scaling covers search and verification methods in more depth.
The paper looked at two methods. One searches over many candidate answers using a separate model that scores each reasoning step. The other lets the model update its answer adaptively for the specific prompt. The key finding is that how well each method works "critically varies depending on the difficulty of the prompt."
That led the authors to a "compute-optimal" strategy that decides how much extra effort to spend on each prompt. They report two results:
- It improved the efficiency of test-time compute scaling by more than 4 times compared to a best-of-N baseline (generate N answers, keep the best one).
- In a comparison with equal total computation, on problems where a smaller base model already had some success, extra test-time compute let it outperform a model 14 times larger.
This is the same budget question as Kaplan and Chinchilla, moved to a new place. Instead of "bigger model or more data," it asks "bigger model or more thinking time per question."
What scaling laws do not tell you
A few limits are worth keeping in mind when you read headlines about them.
- They are fitted curves. Each one holds over the range the authors tested. The 2020 paper says "some trends" span more than seven orders of magnitude, not that they hold forever.
- Loss is not usefulness. The 2020 paper measures loss on text. A lower loss usually means a better model, but it does not tell you directly whether a model is accurate, safe, or good at a specific job. The Chinchilla paper tested downstream tasks for exactly this reason.
- The advice depends on the setup. The two best-known papers gave different budget advice. Newer work adds another dimension, test-time compute, and finds that the best choice depends even on how hard each question is.
The short version
- Scaling laws are curves, found by experiment, that predict how model quality improves with more parameters, data, and compute.
- The 2020 Kaplan paper found smooth power-law trends and advised spending most of a budget on bigger models.
- The 2022 Chinchilla paper found that models should grow with their data, and showed a 70B model beating models up to 530B.
- Newer research applies the same idea to the time a model spends answering, and finds that extra thinking can sometimes beat a much larger model.
