Subscribe
AI
6 min read

What Is Pre-Training? How a Language Model Reads the Internet

Pre-training is the first and by far the most expensive phase of building a large language model. A lab takes a neural network with random weights, feeds it hundreds of billions or trillions of tokens of raw text, and trains it on one task: predict the next token. The result is a "base model" that has absorbed the statistics of human writing at a scale no human has read. Everything a frontier model knows before anyone fine-tunes it into an assistant comes from this phase. This post is part of o

What Is Pre-Training? How a Language Model Reads the Internet

Pre-training is the first and by far the most expensive phase of building a large language model. A lab takes a neural network with random weights, feeds it hundreds of billions or trillions of tokens of raw text, and trains it on one task: predict the next token. The result is a "base model" that has absorbed the statistics of human writing at a scale no human has read. Everything a frontier model knows before anyone fine-tunes it into an assistant comes from this phase.

This post is part of our series of explainers for the concepts behind the news. If you want the surrounding pieces, we have covered what attention is, what next-token prediction is, what fine-tuning is, and what a context window is. This one covers the stage that comes before all of them.

A conveyor of printed cards ends in a blank card under a question mark, a mechanical arm inspecting the line

One objective, and no human labels

Supervised machine learning traditionally needs a person to label data: this email is spam, this photo contains a cat. There is not enough labeled text in the world to train a trillion-token model. Pre-training sidesteps the problem with self-supervised learning, a technique in which the ground truth is inferred from the unlabeled data itself. Hide the next word of a sentence and the visible words are the input; the word that actually follows is the label, for free, on every token of every page. As IBM's definition puts it, the model optimizes against a ground truth that the raw data supplies.

The recipe was named by OpenAI researcher Alec Radford and colleagues in the 2018 paper "Improving Language Understanding by Generative Pre-Training", which introduced the term "generative pre-training": pre-train a language model to generate text, then adapt it to tasks. The architecture today is the decoder-only transformer, and the objective is still the 2018 one. When we explained how transformers work, this is the phase in which the weights stop being random.

The data is the web, minus most of it

Pre-training corpora are built mainly from large-scale web crawls. The GPT-3 paper (Brown et al., 2020) is the clearest published example of the era it started: a dataset of roughly 500 billion tokens assembled from six sources, the largest by far a 410-billion-token subset of Common Crawl web pages after filtering, from which the model consumed about 300 billion tokens during training. Books, academic papers and code made up most of the rest.

Raw crawls are full of duplicates, spam and machine-generated junk, so labs spend heavily on curation: quality classifiers, deduplication, filtering heuristics. That pipeline deserves its own explainer; the short version is that pre-training data work is closer to publishing than to labeling, deciding what fraction of the internet is worth a model's attention.

Every pre-trained model also inherits a knowledge cutoff. GPT-3 stopped seeing new text when its training run ended in 2020, and Llama 3.1's model card states a pretraining cutoff of December 2023. A model cannot know about events after its cutoff, which is why news questions get confident nonsense unless something like retrieval fills the gap.

The compute is what makes it "pre"

Pre-training dwarfs every later phase in cost, and the published numbers explain why only a handful of organizations do it. GPT-3, at 175 billion parameters, was estimated at $4.6 million in hardware for the final run, roughly 1024 V100 GPUs for a month. Epoch AI's cost tracker puts frontier training costs at around $2 million in 2020 and nearly $390 million for the largest runs by 2024.

Meta's Llama 3.1 405B is the best-documented modern run: pretrained on over 15 trillion tokens, consuming a cumulative 39.3 million H100 GPU hours, on a cluster of 16,000 GPUs. Meta called it its first Llama trained at that scale. Post-training, the phase that turns the base model into a chatbot, costs a fraction of this. That asymmetry is why base models are released rarely and chat versions frequently: one pre-training run feeds many rounds of comparatively cheap tuning.

A balance scale with a tall stack of books outweighing a small geared machine

How much data per parameter: the Chinchilla correction

For a fixed compute budget, how big should the model be, and how long should it train? The 2022 DeepMind paper "Training Compute-Optimal Large Language Models" (Hoffmann et al., known as the Chinchilla paper after its 70-billion-parameter test model) answered with a rule of thumb: roughly 20 tokens of training data per parameter. Chinchilla itself trained 70 billion parameters on 1.4 trillion tokens and beat Gopher, a 280-billion-parameter model trained on the same compute, which implied Gopher and its era were "undertrained," too big for the text they had seen.

Most frontier labs now knowingly break that rule. Llama 3.1 405B at over 15 trillion tokens ran at roughly 37 tokens per parameter, about twice the Chinchilla ratio, and smaller open models go far higher. The reason is economics on the other side of training. A model trained longer than compute-optimal performs better than a model of the same quality trained less, and quality-matched smaller models are cheaper to run at inference (Sardana & Frankle, 2024). Epoch AI's overtraining survey (Sevilla et al.) calls models past the 20-token mark "overtrained" and notes developers accept it because tokens spent at inference are paid again on every request.

The other constraint is that the well of unique text is finite. Muennighoff et al. (2023) studied data-constrained training directly: repeating the corpus for up to about 4 epochs costs almost nothing in final loss compared with fresh data, but the value of repetition then decays toward zero. Long-training models are therefore built to eat whatever text exists, more than once if needed.

What you have at the end: a base model, not an assistant

A freshly pre-trained model is a completion engine. Prompt it with "The capital of France is" and it continues plausibly. Ask it "What is the capital of France?" and, depending on its data mix, it may complete the question with another question, because nothing in pre-training taught it that questions deserve answers. Base models also have no safety behavior; they predict text, including the text the lab would rather they not produce.

Bridging that gap is the job of the later phases covered in our RLHF explainer: instruction fine-tuning on curated dialogues, then preference optimization or reinforcement learning to shape what the model says and how it says it. The distinction between base and instruction-tuned models is also the distinction behind most benchmark controversies, since a base model and its tuned twin score very differently on the same test.

Two properties of pre-training explain much of what users encounter later. First, the objective rewards plausibility, not accuracy: a completion that looks like the training text wins, which is structural ground under hallucination. Second, capability tracks the training budget, so announcements of new frontier models are, underneath the marketing, announcements of new pre-training runs: who paid for how many GPU hours, on what data, cut off when.

Sources

AILLMExplainers