Before 2022, frontier artificial intelligence labs scaled large language models primarily by increasing parameter counts while keeping training dataset sizes relatively static. OpenAI's 2020 GPT-3 model trained 175 billion parameters on 300 billion tokens. DeepMind's 2021 Gopher trained 280 billion parameters on 300 billion tokens, and Megatron-Turing NLG scaled to 530 billion parameters on 270 billion tokens. These allocation decisions followed early empirical scaling laws that suggested parameter capacity yielded faster returns on loss reduction than additional token volume.
In March 2022, DeepMind published Training Compute-Optimal Large Language Models by Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and colleagues. The paper demonstrated that early scaling laws were flawed due to experimental artifacts in learning rate scheduling. Under a fixed compute budget, model parameters and training tokens should scale in approximately equal proportion. This finding, termed the Chinchilla scaling law, fundamentally altered pre-training budgets across the industry and later paved the way for inference-optimal over-training in modern open-weight architectures.
The Flaw in Early Scaling Laws
In 2020, Jared Kaplan and collaborators at OpenAI published Scaling Laws for Neural Language Models. The study examined how cross-entropy test loss scales with compute budget C, dataset size D, and parameter count N.
Kaplan et al. fitted a power-law relationship suggesting that when compute increases, model parameters should scale substantially faster than dataset size:
N ∝ C^0.73, D ∝ C^0.27
Under this formulation, an eightfold increase in floating-point operations (FLOPs) dictated expanding parameter count by approximately 4.6x while expanding dataset size by only 1.7x. This finding drove frontier organizations to build models exceeding 100 billion parameters that consumed only 300 billion tokens of natural language data.
The DeepMind team identified a critical experimental flaw in Kaplan's empirical setup: the learning rate schedules were held constant or decayed over a fixed step horizon rather than being retuned to match the total token budget of each specific run. In transformer training with cosine learning rate schedules, the learning rate decays to a minimum value (often 10% of peak) at the exact end of training. If a model is evaluated before its learning rate schedule decays, its measured loss is artificially worse than if it had completed a tuned schedule. This artifact penalized runs on larger datasets and caused researchers to systematically underestimate the value of training tokens.
Hoffmann et al. and the Chinchilla Formulation
To determine true compute-optimal allocations, Hoffmann et al. trained over 400 models ranging from 70 million to 16 billion parameters across dataset sizes ranging from 5 billion to over 500 billion tokens. Each run used a cosine learning rate schedule whose decay horizon matched the target token count.
The authors evaluated compute optimality using three complementary empirical approaches:
- IsoFLOP Profiles: Holding the total training FLOP budget constant, the authors varied parameter size and token volume across 15 models per budget. Plotting loss against parameter count produced a distinct U-shaped curve for each FLOP tier. The valley of each curve identified the optimal parameter size for that budget.
- Fitting the Efficient Frontier: The authors took the minimum loss envelope across all training runs for every compute level and fitted power laws directly to the resulting Pareto frontier.
- Parametric Loss Fitting: DeepMind modeled the final cross-entropy loss as a parametric function of parameters N and tokens D:
L(N, D) = E + A / (N^α) + B / (D^β).
In this formulation, E represents the irreducible entropy of natural language text, while the remaining terms capture the reducible loss attributable to finite model capacity and finite training data.
Using non-linear optimization over empirical training trajectories, DeepMind estimated the constants:
E = 1.69natsA = 406.4B = 410.7α = 0.34β = 0.28
Using the standard approximation for dense transformer pre-training compute, where floating-point operations equal approximately six operations per parameter per token (C ≈ 6ND), minimizing the loss subject to a fixed FLOP constraint C yields:
N_opt(C) = G * (C / 6)^a, D_opt(C) = G^(-1) * (C / 6)^b
where G = (αA / βB)^(1 / (α + β)), a = β / (α + β) ≈ 0.45, and b = α / (α + β) ≈ 0.55.
Because exponents a and b are both close to 0.5, optimal parameter count and optimal token volume scale in roughly equal proportion. Doubling the training compute budget requires increasing both model parameters and training tokens by approximately 41% (2^0.5 ≈ 1.414).
This relationship established the practical rule of thumb for compute-optimal training: an optimal density of approximately 20 tokens per parameter (D / N ≈ 20).
To prove this empirically, DeepMind trained Chinchilla, a 70-billion-parameter model trained on 1.4 trillion tokens, utilizing the exact same compute budget as Gopher (280 billion parameters on 300 billion tokens). Chinchilla decisively outperformed Gopher, GPT-3, Megatron-Turing NLG, and Jurassic-1 across downstream evaluations, including MMLU, GSM8k, and BIG-bench, while requiring four times less memory to serve during inference:
- GPT-3 (OpenAI): 175B parameters, 300B tokens (1.71 tokens/param), 3.15e23 FLOPs, 43.9% MMLU score.
- Gopher (DeepMind): 280B parameters, 300B tokens (1.07 tokens/param), 5.04e23 FLOPs, 60.0% MMLU score.
- MT-NLG (NVIDIA/Microsoft): 530B parameters, 270B tokens (0.51 tokens/param), 8.58e23 FLOPs, 54.4% MMLU score.
- Chinchilla (DeepMind): 70B parameters, 1.4T tokens (20.0 tokens/param), 5.76e23 FLOPs, 67.6% MMLU score.
From Compute-Optimal to Inference-Optimal
While Chinchilla established how to minimize pre-training loss for a fixed training budget, production deployment introduced a different economic reality: pre-training compute is paid once, whereas inference compute is paid continuously on every user request.
During inference, generating an output token requires approximately two floating-point operations per parameter (C_infer ≈ 2N). Serving a 70B model costs four times less compute per token and requires four times less high-bandwidth memory than serving a 280B model. If an organization expects to serve hundreds of billions or trillions of tokens over a model's operational lifecycle, total cost of ownership is dominated by inference, not pre-training.
As formalized in studies like Sardana and Frankle's Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws, the total compute equation across training and serving is:
C_total = C_train + C_infer = 6N * D_train + 2N * D_infer
When projected query demand D_infer is large, the cost-minimizing strategy is to deliberately over-train a smaller parameter model far past its Chinchilla-optimal point.
Meta adopted this inference-optimal strategy with the LLaMA architecture series:
- LLaMA 1 (2023): Trained a 7B model on 1.0 trillion tokens (143 tokens/parameter) and a 65B model on 1.4 trillion tokens (22 tokens/parameter), detailed in the LLaMA technical report.
- LLaMA 2 (2023): Scaled pre-training to 2.0 trillion tokens across all variants, pushing the 7B model to 286 tokens/parameter, as documented in the Llama 2 report.
- LLaMA 3 (2024): Trained both 8B and 70B models on over 15 trillion tokens. For the 8B model, this yielded 1,875 tokens per parameter, nearly 100x the Chinchilla-optimal ratio, as detailed in the Llama 3 technical overview.
Meta's empirical logs demonstrated that even after 15 trillion tokens, test loss for the 8B model continued to decrease monotonically with log-linear improvements, showing no signs of severe degradation or plateau.
Frontiers and the Data Ceiling
As pre-training regimens pushed token-to-parameter ratios beyond 1,000:1, foundation model development ran into a physical constraint: the availability of high-quality human-authored text. Estimates of the total accessible, deduplicated public web text suitable for training range between 15 trillion and 30 trillion tokens.
To continue scaling within inference-constrained parameter footprints, labs deploy three primary mitigation techniques:
- Multi-Epoch Training with Strategic Perturbation: While early deep learning folklore warned against repeating data to avoid overfitting, empirical studies such as Muennighoff et al.'s Scaling Data-Constrained Language Models proved that models can train for 4 to 8 epochs over high-quality data with negligible degradation, provided aggressive deduplication and regularized data ordering are applied.
- Synthetic Data and Model-Filtered Pipelines: Frontier pre-training corpora increasingly rely on synthetically generated reasoning chains, execution-verified code samples, and web text rewritten by frontier models into textbook-style prose.
- Test-Time Compute Expansion: Instead of attempting to compress all capabilities solely into pre-training weights, modern reasoning systems allocate compute dynamically during inference through search, verification, and extended reasoning traces.
Chinchilla scaling proved that model architecture cannot be divorced from data volume. The evolution from Kaplan's parameter-heavy models to DeepMind's compute-optimal balance, and ultimately to modern inference-optimal over-training, reflects an industry maturing from raw parameter counts to balanced hardware efficiency.
Sources
- Scaling Laws for Neural Language Models (Kaplan et al., 2020)
- Training Compute-Optimal Large Language Models (Hoffmann et al., DeepMind 2022)
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (Sardana & Frankle, 2023)
- Scaling Data-Constrained Language Models (Muennighoff et al., 2023)
- LLaMA: Open and Efficient Foundation Language Models (Touvron et al., Meta 2023)
- The Llama 3 Herd of Models (Meta AI, 2024)
