This is an opinion piece. The facts below come from the linked sources; the argument is ours.
If you want one idea that explains why AI got so good so fast, it is a short essay by computer scientist Rich Sutton called The Bitter Lesson. Its claim is blunt: over the long run, general methods that use more computing power beat methods built on human cleverness. Our view is that Sutton is right about the direction of AI progress, and that a popular reading of his essay, that you should just make models bigger, is wrong. The research that came after shows that how you spend computing power, and what data you feed it, still decides who wins.
What the Bitter Lesson says
Sutton's essay opens with its thesis: "general methods that leverage computation are ultimately the most effective, and by a large margin." His reason is that the cost of computing keeps falling, so a method that can use more of it keeps getting better, while a method built on hand-written human knowledge eventually stalls.
He backs this with history. In chess, the program that beat world champion Garry Kasparov in 1997 relied on what Sutton calls "massive, deep search," not on encoding how grandmasters think. Computer Go followed the same path, and so did speech recognition, where statistical methods beat systems built on human knowledge of words and sounds. Computer vision went the same way. In his summary, building in human knowledge always helps "in the short term," but "in the long run it plateaus and even inhibits further progress."
He names the two approaches that keep scaling: "search and learning." Today's chatbots are a learning story. They are trained on huge amounts of text with large amounts of computing power, not programmed with rules about grammar or facts.
Where the popular reading goes wrong
The lazy version of the lesson is that model size is all that matters. A 2022 study from DeepMind, Training Compute-Optimal Large Language Models, is the clearest evidence against that version.
The researchers trained more than 400 language models of different sizes on different amounts of text. They found that many large models of that time were "significantly undertrained." Their explanation was that the field had focused on making models bigger while "keeping the amount of training data constant." Their rule of thumb: for a fixed computing budget, model size and training data should grow in equal proportion, so "for every doubling of model size the number of training tokens should also be doubled." According to the Chinchilla paper, earlier guidance from a 2020 study had said that with 10 times more computing power, a model should grow 5.5 times while its training data grew only 1.8 times. Our explainer on AI scaling laws walks through both papers.
Then they tested it. DeepMind already had Gopher, a model with 280 billion parameters (the internal settings a model learns) trained on 300 billion tokens (small chunks of text). With the same computing budget, the team trained Chinchilla: a model one-quarter the size, at 70 billion parameters, trained on 1.4 trillion tokens. That is more than 4.6 times as much text.
The smaller model won. On the MMLU test of exam-style questions, Chinchilla averaged 67.6% against Gopher's 60.0%, and it did better on 51 of the 57 subjects. The paper says it beat Gopher and even larger models "on nearly every measured evaluation task." Because it is smaller, it also costs less to run every time someone uses it.
Why both things are true
Chinchilla does not refute Sutton. It is still a general method using a lot of computing power, with no hand-written rules. What it refutes is the idea that the only decision left for humans is "more." The same budget produced a better model because people made a better choice about how to split it between size and data.
The paper also points at the next limit. Its authors wrote that "larger, high quality datasets will play a key role in any further scaling of language models." They were careful about how sure they were. Their wording was "Speculatively, we expect that scaling to larger and larger datasets is only beneficial when the data is high-quality," and they noted they had "only two comparable training runs at large scale." That hedge matters, but the direction is clear: once data is part of the recipe, its quality becomes a human decision that computing power alone does not make.
So human judgment did not disappear from AI. It moved. Researchers once spent their time writing down how they thought experts think. Now the hard choices are how to divide a computing budget, what text to train on, and how to measure the result.
Who wins and who loses
The winners are the teams that combine scale with discipline. Access to computing power is the entry ticket, which is why fights over chips and software, like the one we covered in our piece on Amazon's challenge to Nvidia, matter so much. But the Chinchilla result shows a team can get more from the same budget by spending it well, and get a model that is cheaper to serve.
The losers are anyone who treats parameter count as a quality score. That includes buyers. A bigger number on a spec sheet does not tell you a model is better at your task, the same way a leaderboard rank tells you less than you think. In our view, it is also one more reason that, when a smaller, better-trained model can match a bigger one, loyalty to any single model gets hard to justify.
There is a fair counterpoint. Sutton would likely say that choosing a data mix and a model size is exactly the kind of short-term human tuning that the next general method will make unnecessary. He may be right. His essay argues we should build "meta-methods" that discover things for us, not systems that contain what we have discovered. But until those methods exist, the gap between a well-spent budget and a wasted one is real, and it is measured in billions of parameters.
What this means for you
- Do not judge a model by its size. Ask how it performs on tasks like yours.
- Expect smaller models to keep getting better. The Chinchilla result shows that better training can beat raw size.
- Treat data claims seriously. When a company talks about the quality of its training data, that is not a side detail. It is one of the main levers left.
The bottom line: the Bitter Lesson is the best short explanation of why general learning beat hand-built AI. But "use more computing power" was never the whole recipe. The teams that win are the ones that also know how to spend it.
