Chinchilla Scaling Laws: Mathematical Foundations of Compute-Optimal Pre-Training, IsoFLOP Loss Profiles, Parametric Power Laws, and Data-Compute Allocation

When allocating a fixed computational budget to train an autoregressive Transformer, engineers face a fundamental trade-off: should FLOPs be spent increasing the model parameter count ($N$), or should they be spent streaming a larger volume of training tokens ($D$)?

For several years, frontier AI development followed the empirical scaling laws established by Kaplan et al. (2020) from OpenAI. Kaplan's work suggested that model parameter size should scale substantially faster than dataset size ($N \propto C^{0.73}$ versus $D \propto C^{0.27}$), leading organizations to train massive architectures on relatively modest token counts. This paradigm produced models such as GPT-3 (175B parameters on 300B tokens), Gopher (280B parameters on 300B tokens), and Megatron-Turing NLG (530B parameters on 270B tokens).

In 2022, Hoffmann et al. (2022) at DeepMind identified structural measurement flaws in Kaplan's empirical setup and demonstrated that for compute-optimal training, model size and token volume must scale in approximately equal proportion ($N \propto C^{0.5}$ and $D \propto C^{0.5}$). DeepMind validated this hypothesis by training Chinchilla, a 70B parameter model trained on 1.4 trillion tokens using the exact same compute budget as Gopher ($5.76 \times 10^{23}$ FLOPs). Chinchilla outperformed Gopher, GPT-3, and MT-NLG across downstream reasoning, language modeling, and academic benchmarks while requiring one-fourth the inference footprint.

This guide provides an end-to-end mathematical derivation of compute-optimal scaling laws, details the three empirical methodologies used by Hoffmann et al., explains the root causes of the Kaplan discrepancy, explores the economics of post-Chinchilla inference over-training, and provides a production-ready numerical implementation for fitting IsoFLOP curves and solving compute-optimal resource allocations.

1. Computational Budget Modeling: The 6ND Formulation

Before deriving compute optimality, we must establish the mathematical relationship between floating-point operations (FLOPs), model parameters ($N$), and dataset token volume ($D$).

1.1 Forward and Backward FLOP Accounting

In a standard dense decoder-only Transformer, the computational cost of training is dominated by matrix multiplications in the multi-head attention projections and feed-forward networks (FFN).

For each token processed during the forward pass:

1. Linear Projections in Attention: The input embedding of dimension $d_{\text{model}}$ is projected to Query, Key, Value, and Output matrices ($W_Q, W_K, W_V, W_O$). Each linear transformation with weight matrix $W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$ requires $2 \cdot d_{\text{in}} \cdot d_{\text{out}}$ FLOPs (one multiply and one add per weight).
2. Feed-Forward Layers: Standard FFN blocks with hidden dimension $d_{\text{ffn}} = 4 d_{\text{model}}$ require two matrix multiplications ($W_{\text{gate/up}}$ and $W_{\text{down}}$), consuming $2 \times (2 \cdot d_{\text{model}} \cdot 4 d_{\text{model}}) = 16 d_{\text{model}}^2$ FLOPs.
3. Total Forward Pass: Aggregating all matrix operations across $L$ layers yields approximately $2 N$ FLOPs per token, where $N$ is the active non-embedding parameter count:

$$\text{FLOPs}_{\text{forward}} \approx 2 N D$$

During backpropagation, computing the gradients requires two distinct operations for every parameter:

1. Computing gradients with respect to the activations (to propagate error to earlier layers): requires $2 N$ FLOPs per token.
2. Computing gradients with respect to the weight matrices (to update model parameters): requires $2 N$ FLOPs per token.

Thus, the backward pass requires approximately twice the compute of the forward pass ($4 N$ FLOPs per token). Summing forward and backward passes yields the standard total compute budget $C$:

$$C = \text{FLOPs}_{\text{forward}} + \text{FLOPs}_{\text{backward}} \approx 2 N D + 4 N D = 6 N D$$

+--------------------------------------------------------------------------+
|                  Transformer Training Compute Decomposition               |
|                                                                          |
|  [ Forward Pass ]    2 FLOPs / param / token  ───►  2ND FLOPs            |
|  [ Backward Act ]    2 FLOPs / param / token  ───►  2ND FLOPs            |
|  [ Backward Weight]  2 FLOPs / param / token  ───►  2ND FLOPs            |
|  ----------------------------------------------------------------------  |
|  Total Compute Budget (C)                     ───►  6ND FLOPs            |
+--------------------------------------------------------------------------+

1.2 Context Length and Quadratic Attention Overhead

The $6ND$ approximation assumes that linear layers dominate compute. The sequence self-attention operator introduces an additional context-dependent term:

$$\text{FLOPs}_{\text{attn}} = 12 L \cdot S^2 \cdot d_{\text{model}} = 12 L \cdot S \cdot d_{\text{model}} \cdot S$$

where $S$ is the context sequence length. When $S \ll d_{\text{model}}$, attention FLOPs represent less than 1% of total compute. However, when scaling to long-context regimes ($S \ge 32\text{k}$), total compute expands to $C \approx 6 N D + 12 L D S$, requiring explicit attention tiling optimizations such as FlashAttention or RingAttention.

2. The Parametric Loss Surface

Hoffmann et al. (2022) modeled the cross-entropy test loss $L(N, D)$ as a continuous bivariate function composed of an irreducible entropy floor and two power-law penalty terms:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

where:

• $E$ represents the irreducible loss (the inherent Shannon entropy of the natural language data distribution). Even with infinite parameters and infinite data, $L(N, D) \to E$.
• $\frac{A}{N^\alpha}$ represents the parameter capacity bottleneck. A finite model capacity limits the complexity of the learned distribution. As $N \to \infty$, this penalty decays to zero with exponent $\alpha$.
• $\frac{B}{D^\beta}$ represents the data volume bottleneck. Training for a finite number of steps on limited samples prevents the optimizer from reaching the theoretical minimum risk. As $D \to \infty$, this penalty decays to zero with exponent $\beta$.
• $A, B, E, \alpha, \beta$ are strictly positive empirical constants.

                  Parametric Loss Decomposition
                  
      L(N, D) = E + ( A / N^alpha ) + ( B / D^beta )
                │          │                 │
                │          │                 └─ Data Bottleneck
                │          └─ Parameter Bottleneck
                └─ Irreducible Entropy Floor (Natural Language Limit)

3. Mathematical Derivation of Compute-Optimal Scaling

To determine the optimal allocation of parameters $N$ and tokens $D$ for a fixed compute budget $C$, we solve a constrained optimization problem.

3.1 Formulation as Constrained Optimization

We seek to minimize the loss $L(N, D)$ subject to the compute constraint $C = 6ND$:

$$\min_{N, D} L(N, D) = E + A N^{-\alpha} + B D^{-\beta} \quad \text{subject to} \quad 6ND = C$$

From the constraint, we express data volume $D$ as a function of compute budget $C$ and parameter count $N$:

$$D = \frac{C}{6N}$$

Substituting this expression into $L(N, D)$ yields an unconstrained single-variable objective function $L_C(N)$:

$$L_C(N) = E + A N^{-\alpha} + B \left( \frac{C}{6N} \right)^{-\beta} = E + A N^{-\alpha} + B \left( \frac{C}{6} \right)^{-\beta} N^\beta$$

3.2 Finding the Stationary Point

To find the parameter count $N_{\text{opt}}(C)$ that minimizes $L_C(N)$, we take the first derivative with respect to $N$ and set it to zero:

$$\frac{\partial L_C(N)}{\partial N} = -\alpha A N^{-\alpha - 1} + \beta B \left( \frac{C}{6} \right)^{-\beta} N^{\beta - 1} = 0$$

Equating the two terms:

$$\alpha A N^{-\alpha - 1} = \beta B \left( \frac{C}{6} \right)^{-\beta} N^{\beta - 1}$$

Multiplying both sides by $N^{\alpha + 1}$ and dividing by $\beta B \left(\frac{C}{6}\right)^{-\beta}$:

$$\frac{\alpha A}{\beta B} \left( \frac{C}{6} \right)^\beta = N^{\beta - 1 + \alpha + 1} = N^{\alpha + \beta}$$

Taking the $(\alpha + \beta)$-th root of both sides gives the optimal parameter count:

$$N_{\text{opt}}(C) = \left( \frac{\alpha A}{\beta B} \right)^{\frac{1}{\alpha + \beta}} \left( \frac{C}{6} \right)^{\frac{\beta}{\alpha + \beta}}$$

3.3 Derivation of Optimal Token Volume

Using the compute relation $D_{\text{opt}}(C) = \frac{C}{6 N_{\text{opt}}(C)}$, we solve for $D_{\text{opt}}(C)$:

$$D_{\text{opt}}(C) = \frac{C / 6}{\left( \frac{\alpha A}{\beta B} \right)^{\frac{1}{\alpha + \beta}} \left( \frac{C}{6} \right)^{\frac{\beta}{\alpha + \beta}}} = \left( \frac{\beta B}{\alpha A} \right)^{\frac{1}{\alpha + \beta}} \left( \frac{C}{6} \right)^{1 - \frac{\beta}{\alpha + \beta}}$$

Since $1 - \frac{\beta}{\alpha + \beta} = \frac{\alpha + \beta - \beta}{\alpha + \beta} = \frac{\alpha}{\alpha + \beta}$:

$$D_{\text{opt}}(C) = \left( \frac{\beta B}{\alpha A} \right)^{\frac{1}{\alpha + \beta}} \left( \frac{C}{6} \right)^{\frac{\alpha}{\alpha + \beta}}$$

3.4 Power-Law Exponents and The Equipartition Theorem

Let the scaling power-law exponents be defined as:

$$a = \frac{\beta}{\alpha + \beta}, \quad b = \frac{\alpha}{\alpha + \beta}$$

Notice that the sum of the exponents is mathematically invariant:

$$a + b = \frac{\beta}{\alpha + \beta} + \frac{\alpha}{\alpha + \beta} = \frac{\alpha + \beta}{\alpha + \beta} = 1.0$$

Furthermore, we define the baseline prefactor coefficients:

$$G = \left( \frac{\alpha A}{\beta B} \right)^{\frac{1}{\alpha + \beta}}$$

The compute-optimal trajectories simplify to:

$$N_{\text{opt}}(C) = G \left( \frac{C}{6} \right)^a, \quad D_{\text{opt}}(C) = \frac{1}{G} \left( \frac{C}{6} \right)^b$$

+--------------------------------------------------------------------------+
|                  Compute-Optimal Power-Law Relationships                 |
|                                                                          |
|  Optimal Parameters:  N_opt(C) = G * (C / 6)^a                           |
|  Optimal Tokens:      D_opt(C) = (1 / G) * (C / 6)^b                     |
|                                                                          |
|  Exponent Constraint: a + b = 1.0                                        |
|  Equipartition Rule:  If alpha approx beta, then a approx b approx 0.50  |
+--------------------------------------------------------------------------+

4. The Three Empirical Methodologies of Hoffmann et al.

To fit these parameters accurately without bias, Hoffmann et al. (2022) trained over 400 language models spanning 70 million to 16 billion parameters across 5 billion to 500 billion tokens. They evaluated three distinct and complementary fitting approaches.

┌─────────────────────────────────────────────────────────────────────────┐
│                    Three Empirical Estimation Approaches                │
├─────────────────────────┬───────────────────────┬───────────────────────┤
│ Approach 1: Minimum     │ Approach 2: IsoFLOP   │ Approach 3: Bivariate │
│ Training-Curve Envelope │ Slices                │ Parametric Surface    │
├─────────────────────────┼───────────────────────┼───────────────────────┤
│ Evaluates lower convex  │ Holds FLOPs constant; │ Fits L(N,D) over 400+ │
│ envelope across all     │ sweeps N to find      │ final run checkpoints │
│ intermediate points     │ minimum loss parabola │ using Huber loss      │
│ a = 0.50, b = 0.50      │ a = 0.49, b = 0.51    │ a = 0.46, b = 0.54    │
└─────────────────────────┴───────────────────────┴───────────────────────┘

4.1 Approach 1: Minimum Loss Envelope Over Training Runs

• Mechanism: For every model architecture, multiple training runs were executed with varying token counts and learning rate decay horizons. The minimum achieved test loss across all runs at each specific FLOP count was recorded.
• Fitting: A power-law curve was fitted to the lower convex hull of the loss-versus-FLOP envelope.
• Result: $a = 0.50, b = 0.50$.

4.2 Approach 2: IsoFLOP Profiles

• Mechanism: Fixed compute budgets $C_k \in \{6 \times 10^{18}, \dots, 3 \times 10^{21}\}$ FLOPs were established. For each budget, 15 distinct model sizes $N_j$ were trained, with token counts determined exactly by $D_j = C_k / (6 N_j)$.
• Parabolic Fitting: For each IsoFLOP slice, the loss curve $L(N \mid C_k)$ exhibits a distinct U-shape. A parabola was fitted around the minimum to locate the exact empirical optimum $N_{\text{opt}}(C_k)$.
• Result: Power law regression across IsoFLOP minima yielded $a = 0.49, b = 0.51$.

4.3 Approach 3: Parametric Loss Modeling

• Mechanism: All final loss evaluation points across the 400+ runs were fitted directly to the parametric equation $L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$ using the L-BFGS optimization algorithm with a smoothed Huber loss objective to resist outlier contamination:

$$\min_{E, A, B, \alpha, \beta} \sum_{i=1}^M \text{Huber}_\delta \left( \log \left( L(N_i, D_i) - E \right) - \log \left( \frac{A}{N_i^\alpha} + \frac{B}{D_i^\beta} \right) \right)$$

• Fitted Constants:
• $E = 1.6934$
• $A = 406.4$
• $B = 410.7$
• $\alpha = 0.3392 \approx 0.34$
• $\beta = 0.2849 \approx 0.28$
• Result:

$$a = \frac{0.2849}{0.3392 + 0.2849} = 0.456, \quad b = \frac{0.3392}{0.3392 + 0.2849} = 0.544$$

5. Kaplan vs. Hoffmann: Reconciling the Discrepancy

Why did Kaplan et al. (2020) conclude that $N \propto C^{0.73}$ and $D \propto C^{0.27}$, while Hoffmann et al. (2022) found equal $0.5 : 0.5$ scaling?

Rigorous retrospective analyses, including studies by Epoch AI (Porian et al., 2024), identified three primary drivers:

| Discrepancy Driver | Kaplan et al. (2020) | Hoffmann et al. (2022) | Impact on Exponents | | :--- | :--- | :--- | :--- | | Learning Rate Schedule | Fixed 300k-step cosine schedule evaluated early mid-run | Cosine schedule tailored to decay to minimum at step $D_i$ | Early evaluation penalized smaller models on large datasets, depressing $b$ | | Parameter Accounting | Excluded embedding matrix parameters ($N_{\text{non-embed}}$) | Total model parameters including input/output embeddings | Artificially distorted parameter-to-FLOP ratio on smaller models | | Loss Evaluation Metric | Cross-entropy on early tokens before context saturation | Average cross-entropy across full context sequences | Distorted effective loss plateau dynamics |

                       The Scaling Trajectory Divergence
                       
  FLOP Budget (C)       Kaplan Model (Undertrained)      Chinchilla Model (Optimal)
  ─────────────────────────────────────────────────────────────────────────────
  1e22 FLOPs            N = 10.2B,  D = 163B tokens      N = 4.3B,   D = 388B tokens
  1e23 FLOPs            N = 54.8B,  D = 304B tokens      N = 14.6B,  D = 1.14T tokens
  5.76e23 (Gopher)      N = 280B,   D = 300B tokens      N = 70B,    D = 1.40T tokens
  1e24 FLOPs            N = 295B,   D = 565B tokens      N = 49.3B,  D = 3.38T tokens
  1e25 FLOPs            N = 1.58T,  D = 1.05T tokens     N = 166.7B, D = 10.0T tokens

When DeepMind trained Chinchilla (70B parameters, 1.4T tokens) against Gopher (280B parameters, 300B tokens) under the exact same $5.76 \times 10^{23}$ FLOP budget:

• MMLU accuracy increased from 60.0% (Gopher) to 67.5% (Chinchilla).
• GSM8K math accuracy improved from 10.1% to 35.8%.
• Memory required to host the model dropped from 560 GB (7 $\times$ A100 80GB) to 140 GB (2 $\times$ A100 80GB), cutting inference serving costs by over 70%.

6. The Post-Chinchilla Era: Inference Economics and Over-Training

While Chinchilla established the compute-optimal frontier for pre-training, it solves an incomplete optimization problem for production systems. Chinchilla optimality minimizes training compute cost for a target loss, but assumes zero downstream inference queries.

6.1 The Total Lifecycle Cost Equation

In production deployment, the true cost function includes both pre-training and serving inference over $T_{\text{infer}}$ tokens across the lifetime of the model:

$$C_{\text{lifecycle}} = C_{\text{train}} + C_{\text{infer}} = 6 N D_{\text{train}} + 2 N T_{\text{infer}}$$

If a model will be queried over hundreds of billions of inference tokens ($T_{\text{infer}} \gg D_{\text{train}}$), the operational cost is dominated entirely by $2 N T_{\text{infer}}$. Under this regime, minimizing lifecycle cost requires intentionally over-training smaller models far past the Chinchilla boundary.

       Training vs. Lifecycle Optimality
       
       Loss ▲
            │           Chinchilla Optimal (Min Training FLOPs)
            │           [ N = 70B, D = 1.4T ]
            │              \
            │               \     Inference-Optimized Over-Training
            │                \    [ N = 8B, D = 15T ] (LLaMA-3 Regime)
            │                 \      │
            │                  ▼     ▼
            └───────────────────────────────────► Total Lifecycle Cost

6.2 The LLaMA and Open-Weights Paradigm

Meta demonstrated this principle with LLaMA 1 (Touvron et al., 2023) and Llama 3 (Dubey et al., 2024):

• Chinchilla Optimal for 8B Parameters: $D_{\text{opt}} \approx 160\text{B tokens}$.
• Llama 3 8B Actual Training: $D_{\text{train}} = 15\text{T tokens}$ (a $93.75\times$ over-training ratio).
• Even after 15 trillion tokens, the cross-entropy loss for Llama 3 8B continued to decline monotonically with zero signs of empirical degradation or overfitting, proving that the parameter capacity floor ($A / N^\alpha$) was not yet saturated.

7. Extended Frontiers: MuP, Data Repetition, and Test-Time Compute

Scaling laws have expanded beyond dense pre-training into architectural initialization, dataset quality, and test-time reasoning.

7.1 Maximal Update Parameterization ($\mu$P)

Standard initialization (PyTorch default / Xavier) causes optimal learning rates to shrink toward zero as model width $d_{\text{model}} \to \infty$, making hyperparameter sweeping on small models inapplicable to large models. Yang et al. (2022) introduced Maximal Update Parameterization ($\mu$P), which scales weight initializations and layer multipliers by $1/d_{\text{model}}$ such that feature representations update by $\Theta(1)$ at every width. This allows exact zero-shot transfer of optimal learning rates and schedules from a 10M parameter proxy to a 100B+ parameter production model.

7.2 Data-Constrained Scaling Laws

When token budgets exceed unique clean internet text (estimated at 15–30T tokens), models must repeat data. Muennighoff et al. (2023) established that training for up to 4 epochs on repeated tokens yields nearly identical loss reductions to unique data ($D_{\text{eff}} \approx D_{\text{unique}} + 0.9 D_{\text{repeated}}$), but gains decay exponentially after 8–16 epochs due to gradient memorization.

7.3 Test-Time Compute Scaling

Reasoning models (such as OpenAI o1/o3 and DeepSeek-R1) introduce an orthogonal scaling dimension: test-time compute. Snell et al. (2024) showed that spending inference compute on Monte Carlo Tree Search (MCTS), step-level Process Reward Model (PRM) verification, and extended chain-of-thought trajectories scales test performance following power laws that can compensate for more than an order of magnitude in parameter capacity.

8. Production Python Implementation: IsoFLOP Solver & Parametric Fitting

Below is a complete, standalone Python implementation using scipy.optimize and numpy that:

1. Implements the bivariate parametric loss surface.
2. Derives compute-optimal parameter and token allocations for any given FLOP budget.
3. Generates IsoFLOP slices and locates their parabolic minima.
4. Fits the empirical parameters from synthetic or recorded training checkpoint logs.

"""
Chinchilla Scaling Laws: Numerical Solver and IsoFLOP Curve Analyzer.
Implements bivariate loss modeling, constrained optimization, and parametric fitting.
"""

from dataclasses import dataclass
from typing import Tuple, List
import numpy as np
from scipy.optimize import minimize, curve_fit


@dataclass
class ScalingParameters:
    """Parametric loss coefficients from Hoffmann et al. (2022)."""
    E: float = 1.6934   # Irreducible loss (entropy floor)
    A: float = 406.4    # Model size penalty factor
    B: float = 410.7    # Data size penalty factor
    alpha: float = 0.3392  # Model power law exponent
    beta: float = 0.2849   # Data power law exponent

    @property
    def a(self) -> float:
        """Compute-optimal parameter scaling exponent."""
        return self.beta / (self.alpha + self.beta)

    @property
    def b(self) -> float:
        """Compute-optimal token scaling exponent."""
        return self.alpha / (self.alpha + self.beta)

    @property
    def G(self) -> float:
        """Optimal parameter prefactor coefficient."""
        return ((self.alpha * self.A) / (self.beta * self.B)) ** (1.0 / (self.alpha + self.beta))


class ChinchillaSolver:
    def __init__(self, params: ScalingParameters = None):
        self.params = params or ScalingParameters()

    def loss(self, N: np.ndarray, D: np.ndarray) -> np.ndarray:
        """
        Evaluate the parametric loss: L(N, D) = E + A / N^alpha + B / D^beta
        """
        return self.params.E + (self.params.A / (N ** self.params.alpha)) + (self.params.B / (D ** self.params.beta))

    def compute_optimal_allocation(self, FLOPs: float) -> Tuple[float, float, float]:
        """
        Calculate analytical compute-optimal N and D for a given FLOP budget C.
        Returns: (N_opt, D_opt, predicted_loss)
        """
        C_norm = FLOPs / 6.0
        N_opt = self.params.G * (C_norm ** self.params.a)
        D_opt = (1.0 / self.params.G) * (C_norm ** self.params.b)
        min_loss = float(self.loss(np.array([N_opt]), np.array([D_opt]))[0])
        return N_opt, D_opt, min_loss

    def generate_isoflop_curve(self, FLOPs: float, n_points: int = 50) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
        """
        Generate an IsoFLOP slice holding C constant while sweeping N.
        Returns: (N_array, D_array, loss_array)
        """
        N_opt, _, _ = self.compute_optimal_allocation(FLOPs)
        # Sweep parameter range from 0.2x to 5.0x of optimal N
        N_array = np.logspace(np.log10(N_opt * 0.15), np.log10(N_opt * 6.0), n_points)
        D_array = FLOPs / (6.0 * N_array)
        loss_array = self.loss(N_array, D_array)
        return N_array, D_array, loss_array

    def fit_parametric_surface(
        self, 
        N_data: np.ndarray, 
        D_data: np.ndarray, 
        loss_data: np.ndarray
    ) -> ScalingParameters:
        """
        Fit parametric constants (E, A, B, alpha, beta) from empirical runs using L-BFGS-B.
        """
        def objective(p):
            E_val, A_val, B_val, alpha_val, beta_val = p
            pred = E_val + (A_val / (N_data ** alpha_val)) + (B_val / (D_data ** beta_val))
            # Huber loss to provide robustness against outlier runs
            residual = pred - loss_data
            delta = 0.01
            huber = np.where(np.abs(residual) < delta, 0.5 * (residual ** 2), delta * (np.abs(residual) - 0.5 * delta))
            return np.sum(huber)

        # Initial parameter estimates and positivity bounds
        initial_guess = [1.7, 400.0, 400.0, 0.34, 0.28]
        bounds = [(0.5, 3.0), (10.0, 2000.0), (10.0, 2000.0), (0.1, 0.9), (0.1, 0.9)]

        result = minimize(objective, initial_guess, method="L-BFGS-B", bounds=bounds)
        if not result.success:
            raise RuntimeError(f"Surface fitting failed: {result.message}")

        E_fit, A_fit, B_fit, alpha_fit, beta_fit = result.x
        return ScalingParameters(E=E_fit, A=A_fit, B=B_fit, alpha=alpha_fit, beta=beta_fit)


def print_scaling_table():
    solver = ChinchillaSolver()
    budgets = [
        ("Small Scale (1e21 FLOPs)", 1e21),
        ("Medium Scale (1e22 FLOPs)", 1e22),
        ("GPT-3 Match (3.14e23 FLOPs)", 3.14e23),
        ("Gopher / Chinchilla (5.76e23 FLOPs)", 5.76e23),
        ("Frontier Tier (1e25 FLOPs)", 1e25),
        ("Ultra-Scale (1e26 FLOPs)", 1e26),
    ]

    print("==========================================================================================")
    print(f"{'Compute Budget':<35} | {'N_opt (Params)':<15} | {'D_opt (Tokens)':<15} | {'Min Loss':<10}")
    print("==========================================================================================")
    for name, budget in budgets:
        N_opt, D_opt, loss = solver.compute_optimal_allocation(budget)
        
        # Formatting for readability
        n_str = f"{N_opt / 1e9:.2f}B" if N_opt >= 1e9 else f"{N_opt / 1e6:.1f}M"
        d_str = f"{D_opt / 1e12:.2f}T" if D_opt >= 1e12 else f"{D_opt / 1e9:.1f}B"
        
        print(f"{name:<35} | {n_str:<15} | {d_str:<15} | {loss:.4f}")
    print("==========================================================================================")


if __name__ == "__main__":
    solver = ChinchillaSolver()
    print("Scaling Law Exponents:")
    print(f"  Parameter Scaling Exponent (a): {solver.params.a:.4f}")
    print(f"  Token Scaling Exponent (b):     {solver.params.b:.4f}")
    print(f"  Sum of Exponents (a + b):       {solver.params.a + solver.params.b:.4f}")
    print(f"  Prefactor Coefficient (G):      {solver.params.G:.4f}\n")
    
    print_scaling_table()
    
    # Verify IsoFLOP minimum location
    C_test = 5.76e23
    N_arr, D_arr, L_arr = solver.generate_isoflop_curve(C_test, n_points=100)
    idx_min = np.argmin(L_arr)
    print(f"\nIsoFLOP Sweep Verification (Compute = 5.76e23 FLOPs):")
    print(f"  Empirical Minimum at N = {N_arr[idx_min] / 1e9:.2f}B parameters")
    print(f"  Corresponding D = {D_arr[idx_min] / 1e12:.2f}T tokens")
    print(f"  Minimum Test Loss = {L_arr[idx_min]:.4f}")

9. Summary and Practical Engineering Takeaways

1. Equipartition Principle: Under pure pre-training compute constraints, model capacity ($N$) and data volume ($D$) should scale in nearly equal $0.5 : 0.5$ proportions. For every doubling of model parameters, training tokens must also double.
2. Historical Under-Training: Models developed prior to 2022 (GPT-3, Gopher, MT-NLG) were severely undertrained by a factor of $3\times$ to $5\times$ relative to their parameter capacity due to flawed scaling law extrapolation.
3. Inference Amortization Shifts Optimum: When deploying high-traffic production models, minimizing lifecycle cost justifies significant over-training (up to $100\times$ beyond the Chinchilla boundary), as smaller models drastically reduce serving VRAM, KV cache memory footprint, and inference latency.
4. Context FLOP Accounting: Standard $6ND$ scaling holds for short contexts, but quadratic attention compute ($12 L D S$) becomes significant when scaling to sequences of 32k tokens and beyond.

Sources

• Kaplan et al. (2020) - Scaling Laws for Neural Language Models
• Hoffmann et al. (2022) - Training Compute-Optimal Large Language Models
• Brown et al. (2020) - Language Models are Few-Shot Learners (GPT-3)
• Rae et al. (2021) - Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• Smith et al. (2022) - Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B
• Touvron et al. (2023) - LLaMA: Open and Efficient Foundation Models
• Dubey et al. (2024) - The Llama 3 Herd of Models
• Yang et al. (2022) - Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
• Muennighoff et al. (2023) - Scaling Data-Constrained Language Models
• Snell et al. (2024) - Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Porian et al. (2024) - Reconciling Kaplan and Chinchilla Scaling Laws