A large language model writes its answer one token at a time. Each token needs a full pass through the model, and the next pass cannot start until the last token is chosen. That is why long answers from big models feel slow even on fast hardware. In late 2022 and early 2023, two teams at Google and DeepMind described a way to get several tokens out of one pass of the big model, without changing what the model would have written. It is called speculative decoding, or speculative sampling.
This explainer covers why generation is slow, how the draft-and-check trick works, what the two papers measured, and where it stops helping. You do not need to read math to follow it. The formulas are in the Google paper and the DeepMind paper.
Why generating text is slow
Training can process a whole passage in parallel. Generation cannot. To write a reply, the model picks one token, adds it to the text so far, and runs again to pick the next one. The Google paper puts it plainly: decoding K tokens takes K serial runs of the model.
The surprising part is what limits each run. Both papers say the bottleneck is often not arithmetic. It is memory bandwidth, meaning how fast the chip can move the model's weights and its cache of earlier tokens from memory to the compute units. The DeepMind paper notes that for a given set of hardware, the time to generate one token is roughly proportional to the size of the model's parameters and memory. The model size divided by the total memory bandwidth sets a hard ceiling on how fast ordinary token-by-token generation can go.
That has an odd side effect. When a run is limited by memory movement, the chip's arithmetic units are partly idle. Checking several tokens in one run costs almost the same time as producing one token, because the weights are read from memory only once either way. Speculative decoding uses that spare capacity.
If you want background on what tokens are, see our explainer on what tokens are and how usage is billed.
The idea: a fast draft, a careful check
Speculative decoding uses two models. The target model is the big, slow one whose answers you actually want. The draft model is a much smaller, faster model that guesses what the target would say next.
One round works like this:
- The draft model quickly writes a short run of guesses, a few tokens long.
- The target model reads the prompt plus all of those guesses in a single pass. That one pass gives it a verdict on every guessed position at once.
- Going left to right, the system keeps each guess the target model agrees with. At the first guess the target rejects, it throws away the rest of the draft and uses a corrected token from the target model instead.
- If every guess is accepted, the same pass also yields one extra token for free.
So each pass of the big model produces at least one token, the same as normal decoding, and sometimes several. In the Google paper's notation the draft length is called gamma, and a pass yields between 1 and gamma plus 1 tokens. The DeepMind paper describes the same range with its own letter, K.
Think of a junior writer drafting a few words ahead and a senior editor checking them in one glance. When the junior is right, the editor approves several words at once. When the junior is wrong, the editor fixes the first mistake and everything after it is redrawn. The editor never signs off on a word they would not have written.
Why the answers do not change
The key property is that the output is not an approximation. The acceptance rule is a modified form of rejection sampling. For each guessed token, the system compares how likely the target model thought that token was with how likely the draft model thought it was. A token the draft over-proposes is sometimes rejected, and a replacement is drawn from an adjusted distribution. The result, proved in both papers, is that the tokens come out with exactly the probabilities the target model alone would have given.
The Google paper states this as a guarantee that the output distribution is unchanged, and reports "identical outputs" in its experiments. The DeepMind paper makes the same claim with a caveat: the distribution is preserved "within hardware numerics." Different computation graphs produce slightly different floating-point results, so two runs will not match token for token even with the same settings. The authors checked instead that benchmark scores stayed the same, which we cover below.
This is what separates speculative decoding from shortcuts such as a smaller model or heavier compression. Those can trade some quality for speed. Speculative decoding is designed to trade nothing in output quality, and instead spends extra computation to buy lower latency.
Who came up with it
The Google paper, "Fast Inference from Transformers via Speculative Decoding" by Yaniv Leviathan, Matan Kalman and Yossi Matias, introduced the method under the name speculative decoding. The DeepMind paper, "Accelerating Large Language Model Decoding with Speculative Sampling" by Charlie Chen and colleagues, calls it speculative sampling. DeepMind's paper says its work was done concurrently and independently of the Google work, with a heavier focus on serving very large models across many chips, and that the core idea is the same. The Google paper in turn notes that an independent implementation by Chen and colleagues showed similar improvements on Chinchilla 70B.
What the Google paper measured
The Google team tested T5-XXL, an 11 billion parameter model, on two tasks: English to German translation and news summarization. It used smaller T5 models as drafts and measured wall-clock time on a single TPU-v4 chip at batch size 1, for greedy decoding (temperature 0) and standard sampling (temperature 1). The abstract reports a 2X to 3X speedup with identical outputs.
The paper's Table 2 has 12 rows: three draft sizes, two tasks, two temperature settings. Every row ran faster than the baseline. Speedups in the table range from 1.4X to 3.4X.
- Translation, T5-small draft, temperature 0: 3.4X, with 75% of guesses accepted on average.
- Translation, T5-small draft, temperature 1: 2.6X.
- Summarization, T5-small draft, temperature 0: 3.1X.
- Summarization, T5-small draft, temperature 1: 2.3X.
The smallest draft model, T5-small at 77 million parameters, gave the highest speedup in each of the four task and temperature settings, even though larger drafts had higher acceptance rates. With T5-large at 800 million parameters as the draft, translation speedups fell to 1.7X at temperature 0 and 1.4X at temperature 1. A bigger draft guesses better but takes longer to run, and the second cost won. The paper says choosing a draft about two orders of magnitude smaller than the target usually performed best.
Even a trivial draft helps a little. With a bigram model, which only looks at the previous word, as the draft for T5-XXL translation, the paper reports an acceptance rate of about 0.2 and a 1.25X speedup. The paper also suggests simple drafts that copy text from the prompt could work well for tasks like summarization or editing code, where the output often repeats the input. It does not test that idea in its main experiments.
What the DeepMind paper measured
The DeepMind team applied the idea to Chinchilla, a 70 billion parameter model, served across 16 TPU v4 chips. It trained its own 4 billion parameter draft model with 8 layers, built to sample fast on the same hardware. That draft produced a token in 1.8 milliseconds, against 14.1 milliseconds for Chinchilla. All results below are from the paper's Table 1, at batch size 1 with a draft length K of 4.
- XSum summarization, nucleus sampling: 1.92X faster. The ROUGE-2 score was 0.114 with speculative sampling against 0.112 for ordinary sampling.
- XSum summarization, greedy decoding: 2.01X faster. ROUGE-2 was 0.156 against 0.157.
- HumanEval code generation, nucleus sampling: 2.46X faster. The pass rate was 47.0% against 45.1%.
The quality numbers are the point of that table. The XSum scores differ from the baseline by 0.002 and 0.001. On 100-shot HumanEval the pass rate is 1.9 points higher with speculative sampling. The paper reads these results as confirmation that the samples come from the same distribution and that the draft model is not biasing the results, not as a quality gain. The abstract summarizes the speedup as 2 to 2.5 times in a distributed setup.
The paper offers a reason that code gained the most: code has many predictable stretches, such as a loop header, that a small model can guess correctly. It calls this a hypothesis, not a measured result. It also notes that for HumanEval and greedy XSum, the tokens per second exceeded the memory-bandwidth ceiling for ordinary generation, which is the practical sign that the trick is doing something plain decoding cannot.
The tradeoffs
Speculative decoding is not free. The papers point to several limits.
- It spends extra computation. The Google paper says latency improves through increased concurrency at the cost of more arithmetic operations. In its Table 1, an acceptance rate of 0.8 with gamma 5 gave a 3.69X speedup at 1.63X the arithmetic, under the assumption that the draft model costs nothing. When guesses are rejected, that work is wasted.
- It needs spare compute. The paper says the method is not helpful where extra computing resources are not available. Both papers report latency at batch size 1. A busy server that already runs many requests together may have less idle capacity to use. The papers do not report results for that case, so treat the gain there as unmeasured.
- Longer drafts stop paying off. DeepMind's Figure 1 shows that as K grows, the speedup levels off or even gets worse, and the best K for XSum was 3. Later guesses only count if every earlier guess was accepted, and each extra guess costs a draft-model call. Longer drafts also made the time to finish a sequence more variable, which matters if you care about the slowest responses.
- Draft quality matters. Acceptance rates varied by task. The Google paper reports rates between 0.5 and 0.9 for drafts a couple of orders of magnitude smaller than the target, and a few percent for the weakest drafts it tried, a unigram model on a small GPT-like setup.
- You need a compatible draft. DeepMind trained its own 4 billion parameter draft with the same tokenizer and dataset as Chinchilla. Google used existing smaller T5 models as drafts for T5-XXL.
Speculative decoding also combines with other speedups. The DeepMind paper says it does not touch the body of the model, so it can be used alongside techniques such as quantization and multi-query attention. Our explainer on model distillation covers another way to get smaller, faster models, which is also how some draft models are made.
What this means if you use AI models
For most users the effect shows up as speed, not as a feature. If a provider uses speculative decoding, the same model answers sooner, and the answer quality is meant to be the same. Whether a given product uses it, and with which draft model, is up to the provider and is usually not stated, so check the provider's current documentation before assuming it. For the proofs behind the guarantee, see our post on speculative decoding mathematical foundations. For serving trade-offs with other draft methods, see our post on speculative decoding in production serving.
For builders, the useful questions are practical ones. Is your serving setup limited by memory bandwidth at small batch sizes? Do you have a draft model that shares the target's tokenizer? Does your workload have predictable text, such as code or edits to existing text, where guesses are accepted often? The papers suggest the gain depends heavily on those answers, so measure on your own traffic.
The takeaway is simple. Generating text token by token leaves chip capacity unused, and speculative decoding fills that gap with guesses that a bigger model then checks all at once. The two papers measured speedups between 1.4X and 3.4X on models from 11 billion to 70 billion parameters, with output distributions that are provably the same as ordinary decoding.
