Model distillation is how AI labs turn a big, expensive model into a smaller, cheaper one that behaves much like it. A large "teacher" model produces answers, and a smaller "student" model is trained to copy them. The result can run faster and cost less while keeping much of the teacher's skill on the tasks it was trained for.
The idea is more than a decade old. It now sits in the middle of two very different stories: the small models that run on phones and laptops, and a fight between AI labs over who is allowed to learn from whose model. This guide explains how distillation works, where you meet it, what it cannot do, and why labs now treat it as a security problem.
The short version
- Teacher: a large, capable model. It is expensive to run.
- Student: a smaller model. It is cheaper and faster to run.
- Distillation: training the student on the teacher's outputs, so the student learns to answer the way the teacher does.
- Why bother: a smaller model needs less memory and less compute for each answer, so it costs less to serve and can fit on smaller hardware.
- The catch: the student usually gives up some quality, and it learns mainly what the teacher showed it.
Where the idea came from
The method most people cite is a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, Distilling the Knowledge in a Neural Network. Their starting problem was practical. The paper says that averaging the predictions of many models improves results, but that "making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets."
Their answer was to compress what the big model, or group of models, had learned into a single smaller model that is easier to deploy. The paper reports results on MNIST, a standard handwritten digit dataset, and says the authors could "significantly improve the acoustic model of a heavily used commercial system" by distilling an ensemble into one model. The abstract notes that Caruana and his collaborators had already shown that the knowledge in an ensemble can be compressed into one model; Hinton's team developed the approach further with a different technique.
How distillation works: soft targets
To see why copying a teacher helps, it helps to know what a model actually outputs. A classifier does not just say "this is a 2." It gives a probability for every possible answer: very likely a 2, a tiny chance of a 3, an even tinier chance of a 7.
Normal training only cares about the right answer. Hinton's paper argues that the small probabilities on the wrong answers carry useful information too. Its example: "An image of a BMW, for example, may only have a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot." Those ratios tell you how the model sees the world, which things look alike and which do not.
So instead of training the student only on the correct labels, the paper trains it on the teacher's full set of probabilities. It calls these soft targets. In the paper's handwritten digit example, one version of a 2 might get a probability of one in a million of being a 3 and one in a billion of being a 7, while another version of a 2 gets the reverse. That pattern says which 2s look like 3s and which look like 7s.
There is a problem: those numbers are so close to zero that they barely affect training. The paper's fix is a setting called temperature. Raising the temperature of the final step of the model, the softmax, makes its probabilities "softer," so the small ones become large enough to matter. The teacher produces its soft targets at a high temperature, and the student is trained at that same high temperature. After training, the paper says, the student goes back to a temperature of 1 for normal use. When the correct labels are also known, the paper found it helps to combine the soft-target training with a second objective on those true labels.
The data used for this step is called the transfer set. The paper notes it could consist entirely of unlabeled data, or it could be the original training set.
How it works for chatbots and language models
Language models predict the next token, a word or part of a word, by giving a probability to every token in their vocabulary. That makes them a natural fit for the same trick. If you want the basics on tokens first, see our explainer on what tokens are.
In practice, people use two broad versions of distillation for language models.
Matching the teacher's probabilities. If you have full access to the teacher model, the student can learn from the teacher's probability for every possible next token, not just the token the teacher picked. Google used this for its Gemma 2 open models. The Gemma 2 technical report says the team trained "the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction," and says the resulting models "even offer competitive alternatives to models that are 2-3 times bigger."
Training on the teacher's text. If you only have the teacher's finished answers, you can collect a large set of prompts and answers and fine-tune the student on them. This is ordinary supervised fine-tuning, with the teacher's answers as the examples. Researchers studied a version of this for translation in Sequence-Level Knowledge Distillation (Yoon Kim and Alexander Rush, 2016). They report that their best student "runs 10 times faster than its state-of-the-art teacher with little loss in performance," and that adding weight pruning gave a student with 13 times fewer parameters than the teacher, at a cost of 0.4 BLEU, a standard translation score.
The second version matters most for today's debates, because it does not need access to the teacher's internals. Anyone who can send a model prompts and save its answers can build a training set from them.
If you want the math behind both versions, including forward and reverse KL divergence and sequence-level distillation, our technical guide to knowledge distillation in large language models covers it.
Real examples you may have used
DeepSeek-R1's distilled models
DeepSeek's R1 release in January 2025 is a well-documented recent case. In its GitHub repository, DeepSeek says it open-sourced "six dense models distilled from DeepSeek-R1 based on Llama and Qwen," at sizes of 1.5B, 7B, 8B, 14B, 32B and 70B parameters. The repository says these models "are fine-tuned based on open-source models, using samples generated by DeepSeek-R1," and that the Qwen-based versions were "finetuned with 800k samples curated with DeepSeek-R1."
This is distillation by text: the big reasoning model wrote worked answers, and smaller existing models were trained on them. DeepSeek reports that this gave better results than training the small models with reinforcement learning on their own. In the repository's words, "the reasoning patterns of larger models can be distilled into smaller models, resulting in better performance compared to the reasoning patterns discovered through RL on small models." The DeepSeek-R1 paper makes the same point, saying the reasoning patterns of its large models "can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models."
DeepSeek also chose to allow it. The repository says the R1 series supports commercial use and allows "any modifications and derivative works, including, but not limited to, distillation for training other LLMs."
DistilBERT
An earlier example is DistilBERT, described in a 2019 paper. Its authors distilled the BERT language model during pre-training and report that it is "possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster." The paper pitches the smaller model for on-device use.
Distillation as a product feature
Distillation is also something companies sell to developers. In October 2024, OpenAI launched Model Distillation in the API, a workflow that, in OpenAI's words, lets developers "use the outputs of frontier models like o1-preview and GPT-4o to fine-tune and improve the performance of more cost-efficient models like GPT-4o mini." OpenAI describes distillation as "fine-tuning smaller, cost-efficient models using outputs from more capable models, allowing them to match the performance of advanced models on specific tasks at a much lower cost." Its suggested process is to set up an evaluation first, store the big model's answers on real tasks, fine-tune the small model on them, and then use the evaluation to check whether the smaller model is good enough to deploy.
Note the words "on specific tasks." That is the honest framing of distillation for most businesses: a small model trained to do one job about as well as a big one, not a small copy of the whole big model.
Distillation vs quantization, pruning and fine-tuning
Distillation is one of several ways to make AI cheaper to run. They are often confused.
- Distillation trains a new, smaller model to imitate a bigger one. The student can have a different design and far fewer parameters.
- Quantization keeps the same model but stores its numbers with fewer bits, so it takes less memory. Our guide to Q4, Q8 and quantization covers this.
- Pruning removes parts of a model, such as weights that contribute little. The translation study above combined pruning with distillation.
- Fine-tuning is further training of an existing model on new examples. Distillation by text is a kind of fine-tuning in which the examples come from a teacher model.
These methods stack. A model can be distilled from a larger one, then quantized so it fits on a laptop.
What distillation cannot do
The student is usually behind the teacher. The sources above report a student that keeps most, not all, of the teacher's quality: 97% of BERT's language understanding for DistilBERT, and "little loss" for the translation student. Hinton's paper also notes that the small model "cannot exactly match the soft targets."
It learns what it was shown. A student trained on a teacher's answers to coding questions learns coding answers. OpenAI's own description promises teacher-level performance "on specific tasks," which is the realistic goal.
It copies mistakes as well as skills. The student is trained to answer the way the teacher does. Nothing in the method separates the teacher's good habits from its errors. If the teacher makes things up on some kinds of questions, the training data will contain those answers. Our explainer on AI hallucinations covers why models do this.
It needs a teacher. Distillation moves capability from a big model to a small one. It does not, on its own, create a model better than the best teacher available.
Why AI labs fight over distillation
That last point is why distillation became contested. If a lab can build a strong student from another company's model, it can skip part of the cost of building that capability itself. And distillation by text needs nothing more than access to the teacher's answers.
The big closed-model companies forbid this in their terms. OpenAI's Terms of Use for ChatGPT and its other services for individuals list, among the things users may not do, "Use Output to develop models that compete with OpenAI" and "Automatically or programmatically extract data or Output." Anthropic's Commercial Terms say customers may not "access the Services to build a competing product or service, including to train competing AI models," unless Anthropic expressly approves it.
The rules differ from lab to lab. DeepSeek, as noted above, explicitly allows distillation from R1 for training other models. Whether a given use is allowed depends on the license of the model you learn from, which is one of the practical differences between open-weight and closed models.
The dispute is now a recurring news topic. We have covered a report in The Information that DeepSeek and Moonshot AI distilled OpenAI's GPT-6 Astra, Dean Ball, who now works at OpenAI, estimating that distillation explains about half of why Chinese labs are close behind on benchmarks (a personal estimate; China rejects US distillation allegations), and Anthropic's published findings on defenses that it says cut reasoning distillation extractions by 90% to 99%. Each of those posts gives the claims and the responses in full.
The core tension is simple. The same technique is a normal, encouraged engineering tool when a company distills its own models or a developer uses a permitted API workflow, and a terms violation when someone uses a rival's closed model as an unpaid teacher.
What it means for you
Small models got better partly because of distillation. When a compact model that runs on a laptop or phone performs well for its size, distillation from a larger model is often one reason. Gemma 2's 2B and 9B models and DeepSeek's R1 distilled models are documented examples.
"Distilled" on a model name tells you something. A name like DeepSeek-R1-Distill-Qwen-7B means a 7B Qwen model trained on R1's answers. Expect it to be faster and cheaper than R1, and expect it to be weaker than R1.
If you build with AI, it is a cost lever. If one big model handles a narrow, repeated task well, distilling that task into a smaller model is a standard way to cut cost and response time. Test the student against the teacher before you switch, which is the step OpenAI's own workflow builds in.
Read the license. Training on another model's output may break its terms. The OpenAI and Anthropic terms quoted above both restrict using outputs to build competing models.
Sources
- Distilling the Knowledge in a Neural Network (Hinton, Vinyals, Dean, 2015)
- Sequence-Level Knowledge Distillation (Kim and Rush, 2016)
- DistilBERT, a distilled version of BERT (Sanh et al., 2019)
- Gemma 2: Improving Open Language Models at a Practical Size (Google, 2024)
- DeepSeek-R1 paper (arXiv)
- DeepSeek-R1 GitHub repository
- OpenAI: Model Distillation in the API
- OpenAI Terms of Use
- Anthropic Commercial Terms of Service
