Instruction tuning is additional training for a pretrained language model on examples where the input is a task written as a plain-language instruction and the target is a good response to that task. It uses ordinary supervised learning, so the model is trained to predict the target response given the instruction. The result is a model that treats a request as something to carry out rather than as text to continue.
What is instruction tuning?
A pretrained model, often called a base model, learns one skill during pretraining: given some text, predict what comes next. That skill produces fluent completions. It does not by itself produce obedience.
FLAN, "Finetuned Language Models Are Zero-Shot Learners" defines instruction tuning as finetuning language models on a collection of tasks described via instructions. In that work, the authors took a pretrained model with 137B parameters and instruction-tuned it on over 60 NLP tasks written with natural language instruction templates, then tested it on task types it had not seen during tuning.
The idea is literal rather than abstract. You assemble many examples of "here is a request, here is a good answer." You then run more gradient steps on those examples. The model's weights shift so that instruction-shaped input tends to be followed by answer-shaped output.
Why does a base model not follow instructions?
Pretraining text contains questions, but a question in raw text is usually followed by discussion, commentary, or an unrelated next sentence. Nothing in the pretraining signal marks a particular string as a request directed at the model that must be satisfied.
Scale does not fix this on its own. The InstructGPT paper states that making models bigger does not inherently make them better at following a user's intent.
Instruction tuning supplies the missing signal. It shows the model, many times over, that this kind of input is followed by that kind of output.
How does instruction tuning work?
The training data is a set of pairs, sometimes described as triples because a middle field is optional.
- Instruction: the task, written in natural language. For example, a request to convert a sentence into the past tense.
- Input (optional): the specific content the instruction applies to, such as the sentence itself. Some tasks need no separate input because the instruction already contains everything.
- Response: the desired output.
Here is one made-up example, written only to show the shape of the data. Instruction: "Rewrite this sentence in the past tense." Input: "The dog chases the ball." Response: "The dog chased the ball."
Each example is rendered into a single text string using a template, and the model is trained on that string with the standard supervised loss. In other words, the model is trained to predict the target response tokens. Implementations differ in details such as which tokens contribute to the loss (our technical guide to supervised fine-tuning covers loss masking and sequence packing), but the objective is ordinary next-token prediction on a curated dataset rather than a new learning algorithm.
Two properties matter. Many tasks are mixed together, so the model learns to respond to instruction phrasing in general instead of memorizing one task. And the instructions are written in natural language. FLAN was then tested on task types it had not seen during tuning. FLAN's ablations point to the number of finetuning datasets, model scale, and the use of natural language instructions as key to the results.
Nothing about the architecture changes. Instruction tuning reuses the pretrained model and the pretraining machinery, applied to a different dataset.
How is instruction tuning different from RLHF?
Instruction tuning is supervised learning on demonstrations. Each example shows one good response, and the model is trained to produce it.
RLHF is a preference step. Instead of a single target answer, it uses human judgments about which of several model outputs is better, and it updates the model using a reward derived from those judgments.
The InstructGPT pipeline shows how the two connect. The authors collected labeler-written demonstrations of desired behavior on labeler-written prompts and on prompts submitted through the OpenAI API, used them to fine-tune GPT-3 with supervised learning, then collected rankings of model outputs and fine-tuned further with RLHF. That supervised stage on demonstrations is a form of instruction tuning.
So the two are complementary and usually sequential. Demonstrations teach the model what a good answer looks like. Preference data teaches it to choose between plausible answers. Some instruction-tuned models then go through a preference step. Later methods such as DPO solve the same preference problem with a simple classification loss.
The InstructGPT paper also shows how much this post-training can matter. In human evaluations on their prompt distribution, outputs from the 1.3B InstructGPT model were preferred to outputs from 175B GPT-3, despite having 100x fewer parameters.
How much data does instruction tuning need?
Published work spans a wide range, and the papers count it in different units: tasks in FLAN and Flan-PaLM, examples in LIMA.
FLAN used over 60 tasks, and its ablations report that the number of finetuning datasets matters. "Scaling Instruction-Finetuned Language Models" went further on scale, studying the effect of scaling the number of tasks, scaling model size, and finetuning on chain-of-thought data. There, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PaLM 540B by +9.4% on average and reaches 75.2% on five-shot MMLU.
LIMA, "Less Is More for Alignment" went small. The authors fine-tuned a 65B LLaMa model with the standard supervised loss on only 1,000 carefully curated prompts and responses, with no reinforcement learning and no human preference modeling.
These results point two ways. The FLAN and Flan-PaLM work reports gains from more tasks. The LIMA authors argue that a small, carefully chosen set can be enough to teach response format and style. Data quality and data quantity both appear in these papers as levers.
Can a model write its own training data?
Partly, according to Self-Instruct. That paper starts from a practical problem: instruction-tuned models depend on human-written instruction data, which is limited in quantity, diversity, and creativity.
The method has a language model generate instructions, inputs, and outputs. The generated items are filtered to remove invalid ones and ones too similar to items already in the pool. The original model is then finetuned on the filtered synthetic data.
Applied to vanilla GPT-3, Self-Instruct produced a 33% absolute improvement on Super-NaturalInstructions, which the paper reports as on par with InstructGPT-001. In human evaluation on expert-written novel tasks, a 5% absolute gap behind InstructGPT-001 remained. The authors released a large synthetic dataset.
The takeaway is that models can expand the supply and variety of instruction data, while human-written data still contributes something the synthetic pool did not fully reproduce in that evaluation.
What do the studies report?
- FLAN reports that the instruction-tuned 137B model beat zero-shot 175B GPT-3 on 20 of 25 tasks evaluated, and beat few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. (arxiv.org/abs/2109.01652)
- Scaling Instruction-Finetuned Language Models reports that Flan-PaLM 540B, instruction-finetuned on 1.8K tasks, outperforms PaLM 540B by +9.4% on average and reaches 75.2% on five-shot MMLU. The same work released Flan-T5 checkpoints, which the paper reports have strong few-shot performance even compared to much larger models such as PaLM 62B. Its conclusion is that instruction finetuning is a general method for improving the performance and usability of pretrained models. (arxiv.org/abs/2210.11416)
- InstructGPT reports that, in human evaluations on their prompt distribution, outputs from the 1.3B model were preferred to outputs from 175B GPT-3, despite 100x fewer parameters. (arxiv.org/abs/2203.02155)
- Self-Instruct reports a 33% absolute improvement on Super-NaturalInstructions for vanilla GPT-3, on par with InstructGPT-001, with a 5% absolute gap remaining on expert-written novel tasks in human evaluation. (arxiv.org/abs/2212.10560)
- LIMA reports that, in a controlled human study, its responses were equivalent or strictly preferred to GPT-4 in 43% of cases, 58% versus Bard, and 65% versus DaVinci003. The authors conclude that almost all knowledge in large language models is learned during pretraining, and that only limited instruction tuning data is needed to teach high quality output. That is the authors' interpretation of their own results. (arxiv.org/abs/2305.11206)
What are the limits?
Instruction tuning shapes behavior more than it adds knowledge. On the LIMA authors' interpretation, the underlying knowledge comes from pretraining, and the tuning data teaches the model to produce high quality output in the expected format.
Data quality and diversity matter. FLAN's ablations identify the number of finetuning datasets, model scale, and natural language instructions as key factors. Self-Instruct starts from the point that human-written instruction data is limited in quantity, diversity, and creativity.
Synthetic data did not close the whole gap in the Self-Instruct evaluation, where a 5% absolute difference behind InstructGPT-001 remained on expert-written novel tasks.
Evaluation scope is another limit. InstructGPT's human evaluations were run on the authors' prompt distribution, so the results describe behavior on those prompts rather than on every possible use. Similarly, FLAN's comparisons cover the 25 tasks evaluated, not all tasks.
Finally, instruction tuning is often not the last step. In the InstructGPT pipeline, RLHF came after the supervised stage. LIMA, by contrast, used no reinforcement learning or preference modeling at all, so how much a later preference step adds depends on the setup.
Why does this matter if you are not building models?
Chat assistants grew out of this work. OpenAI says it trained ChatGPT with the same methods as InstructGPT, starting with supervised fine-tuning on conversations written by human AI trainers. The conversational behavior you experience, where a request gets an answer rather than a continuation, comes from this training step plus any preference tuning that follows.
When you see an open model released in two variants, one labeled "base" and one labeled "instruct" or "chat," the usual meaning is that the instruct or chat variant has been through instruction tuning, and often through a preference step as well. The base variant has not. In practical terms, a base checkpoint may respond to your prompt by extending it, while the instruct checkpoint attempts to satisfy it.
It also shows that parameter count does not tell the whole story. InstructGPT reports a smaller tuned model preferred over a much larger untuned one, and the scaling work reports Flan-T5 checkpoints performing well against a larger PaLM model. On the LIMA authors' reading, tuning mostly changes how a model presents what it learned in pretraining.
Related explainers
Sources
- Finetuned Language Models Are Zero-Shot Learners (FLAN)
- Scaling Instruction-Finetuned Language Models
- Training language models to follow instructions with human feedback (InstructGPT)
- Self-Instruct: Aligning Language Models with Self-Generated Instructions
- LIMA: Less Is More for Alignment
- OpenAI: Introducing ChatGPT
