What Is Next-Token Prediction? How Language Models Write One Token at a Time
Every large language model on the market, from the smallest open-weights release to the biggest closed API, was built to do one thing: look at a string of text and guess what comes next. That single objective, called next-token prediction, is the closest thing the field has to a founding idea. It is worth understanding properly, because it explains both what LLMs are good at and why they fail the way they do.
The short version
A language model takes the text so far, called the context, and outputs a probability for every possible next token. A token is the unit of text the model works with, typically a word or a fragment of a word. The chosen token gets appended to the context, the model runs again, and the loop continues until generation stops. Chatbots, code assistants, and agents all run on this same loop.
Two things fall out of this framing immediately. First, the model is not looking up answers in a database; it is producing the most statistically defensible continuation of the text in front of it. Second, "understanding" is never part of the objective. What the model does is measure how well a task can be reduced to continuation.

An idea from 1951
The intuition predates the transformers by half a century. In "Prediction and Entropy of Printed English," Claude Shannon measured how information-rich English is by running a guessing game: show a human subject a passage letter by letter and have them guess each next letter, counting the attempts until they get it right. Long-range statistical structure in ordinary English, word patterns, grammar, spelling habits, reduced the entropy to something on the order of one bit per letter, which Shannon tied to a redundancy of roughly 75 percent (Shannon 1951). Prediction, in other words, is a measurement of how much structure has been captured.
Modern language models inherited this directly. The standard training metric, perplexity, is the exponential of average prediction error. A lower perplexity means the model assigns higher probability to the text that actually appears. In 2022 researchers at DeepMind ran Shannon's experiment on steroids, comparing humans and language models head-to-head at guessing the next token in real text. Humans lost: they were consistently worse at next-token prediction than models including GPT-3-Ada (Eccleston et al., 2022). The thing an LLM does better than you is the one thing it was trained to do.
Training needs no labels
Next-token prediction has a property that made it scale: the training signal is free. Any text already on the internet contains its own answer key. Take a sentence, and at every position, the token that follows is the target. The model is asked to score each position in the context window at once, with a causal attention mask preventing any position from peeking at the tokens to its right (Jurafsky & Martin, Speech and Language Processing, ch. 7; Raschka, 2025). Error gradients push probabilities toward whatever the corpus actually says comes next.
This self-supervised setup is what removed the bottleneck of hand-labeled data. GPT-2 trained on 1.5 billion parameters worth of scraped web text using nothing but next-token prediction and still picked up enough to do translation and reading-comprehension questions without ever being trained on them (Radford et al., 2019). GPT-3 scaled the same objective to 175 billion parameters and roughly 300 billion training tokens, with no task-specific fine-tuning at all (Brown et al., 2020; NVIDIA technical overview).
The GPT line of papers reframed every downstream task as continuation. Ask "translate this to French:" and the model's job is to predict the French, because that is what follows such prompts in the training data. Question answering, summarization, and code generation all fold into the same loop. That is what the GPT-2 title "Language Models are Unsupervised Multitask Learners" was claiming, and the follow-on research program of the last six years has largely been stress tests of that sentence.
Generation: the loop, and where temperature lives
At inference time the model is run one position at a time. Each forward pass outputs a probability distribution over the vocabulary, GPT-2's vocabulary contains 50,257 tokens, and the system must pick one. Greedy decoding takes the maximum; modern assistants sample from the distribution, using knobs like temperature and top-p to reshape how flat or peaked the probabilities are during sampling (Jurafsky & Martin, ch. 7). The chosen token joins the context and the loop repeats.

The economics of the loop matter practically. Every generated token costs a full forward pass over the current context, and keeping the key-value cache between steps is what makes this tractable at all. This is why output tokens cost more than input tokens in API pricing and why long generations get slow: autoregressive decoding is inherently sequential.
Why a boring objective turns up producing something useful
The uncomfortable empirical fact of the last five years is that next-token prediction at scale does more than it was asked to. A model minimizing prediction error on web text must implicitly learn grammar, encyclopedic facts, writing style, code syntax, and something resembling logic, because all of those patterns help predict what comes next.
Ilya Sutskever's version of the argument, delivered at a 2024 workshop, was that predicting the next token well "means that you understand the underlying reality that led to the creation of that token," which is why the simple objective feels deceptively deep (R&A transcript; Simons Institute). Researchers who are not convinced point out that pattern completion and understanding are not the same thing, and the disagreement over this exact question is the central argument in AI capability forecasting. Neither side can yet settle it, because the evidence cuts both ways depending on the benchmark.
Where the objective runs out of road
The limitations of LLMs are mostly the limitations of this training goal, visible in its raw form.
Hallucination is the most direct case. The model is rewarded for plausible continuation, not for true continuation. If the statistics of the context say a confident answer is likelier than a hedge, the objective does not punish the confidence. This is why retrieval-augmented generation and tool use exist: they change what goes into the context so that the statistically likely continuation is one backed by evidence.
Long-horizon consistency is a second gap. The model optimizes the next token, never the final document. Plans that fall apart at the tenth step are not a bug against the training objective; they are exactly what one-token-at-a-time optimization predicts. Reinforcement learning over full outputs and test-time methods that search over sequences, tree search or best-of-n sampling, are attempts to move the unit of optimization closer to the unit of the task.
Reasoning is the live frontier. Reasoning models trained with RL on verifiable answers still emit tokens one at a time; the chain of thought is itself just a long sequence of next-token predictions, only now the intermediate text buys extra compute before the final answer is committed. The objective stayed the same; the interesting part is what the model learned to put in the context to help itself.
FAQ
Is next-token prediction the same as autocomplete? Mechanically yes, in capability no. At small scale it is autocomplete; at web scale plus a post-training phase, it is the same loop that powers assistants. The scaling is the point, which is also why "just autocomplete" remains a contested summary rather than a settled one.
Why tokens and not words? Subword tokenization guarantees the model can spell anything, including misspellings and non-English text, using a fixed vocabulary. Words would require an unbounded one.
Does the model know what it is doing between tokens? No. There is no state beyond the context window and the weights. Every run is a fresh probability calculation over the text so far.
Sources
- C. Shannon, "Prediction and Entropy of Printed English," Bell System Technical Journal, 1951. https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
- A. Radford et al., "Language Models are Unsupervised Multitask Learners," OpenAI, 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- T. Brown et al., "Language Models are Few-Shot Learners," 2020. https://arxiv.org/abs/2005.14165
- D. Eccleston et al., "Language models are better than humans at next-token prediction," 2022 (rev. 2024). https://arxiv.org/abs/2212.11281
- J. Jurafsky & D. Martin, Speech and Language Processing, ch. 7: Transformers and Language Models. https://web.stanford.edu/~jurafsky/slp3/7.pdf
- S. Raschka, "How does next-token prediction train a large language model?" Ahead of AI FAQ. https://sebastianraschka.com/faq/docs/next-token-prediction.html
- C. Wolfe, "Language Model Training and Inference: From Concept to Code." https://cameronrwolfe.substack.com/p/language-model-training-and-inference
- NVIDIA, "OpenAI Presents GPT-3, a 175 Billion Parameters Language Model." https://developer.nvidia.com/blog/openai-presents-gpt-3-a-175-billion-parameters-language-model/
- Simons Institute, "Large Language Models Take Center Stage in Simons Institute Workshop." https://simons.berkeley.edu/news/large-language-models-take-center-stage-simons-institute-workshop
- R&A, transcript of Ilya Sutskever on next-token prediction and superhuman AI. https://ea.rna.nl/2023-12-15/what-makes-ilya-sutskever-believe-that-superhuman-ai-is-a-natural-extension-of-large-language-models
