Subscribe
educational
6 min read

What Is Model FLOPs Utilization? The Number That Says If Your GPU Cluster Earned Its Rent

MFU, the efficiency metric introduced by Google's PaLM paper and now quoted in every frontier training report, explained: the 6ND numerator, the sparsity trap in the denominator, what 38-43% actually means, and why nvidia-smi tells you nothing.

What Is Model FLOPs Utilization? The Number That Says If Your GPU Cluster Earned Its Rent

Every serious training run now ships with a quality stamp: the Llama 3 paper says Meta's 405B model was trained at 38 to 43% Model FLOPs Utilization (MFU). The PaLM paper that introduced the metric in 2022 reported 46.2% and put the previous state of the art into comparison: 21.3% for GPT-3.

MFU answers one question: of all the arithmetic your chips could theoretically do, what share went into the math your model actually needs? The rest went to communication, pipeline bubbles, memory stalls, recompute, and idle silicon. Since frontier training budgets run in the tens of millions of GPU-hours, every percentage point of MFU is money.

A row of stylized machine cabinets with flywheels, most stopped, under a gauge dial reading below half

The definition

The PaLM team defined MFU as "the ratio of the observed throughput (tokens-per-second) relative to the theoretical maximum throughput of a system operating at peak FLOPs," with one crucial constraint: the theoretical maximum "only accounts for the required operations to compute the forward+backward passes, and not rematerialization."

Written out:

MFU = (model FLOPs per token x tokens processed per second) / (peak FLOPs per device x number of devices)

The Llama 3 paper uses the equivalent form: model FLOPs per step divided by (seconds per step x number of GPUs x peak FLOPs per GPU). The numerator counts only the math the model logically requires. Every inefficiency hides in the gap: the 60% of an H100 you never see.

The numerator: 6ND

Model FLOPs per token come from arithmetic, not measurement. The standard approximation, used by OpenAI's scaling-law work and adopted by Google and Meta, is that a dense Transformer training step costs about 6N FLOPs per token, where N is the parameter count. Forward pass: each parameter participates in one multiply-accumulate, so 2N. Backward pass: two more matrix products per parameter, one for the weight gradients, one for the input gradients, so 4N. Total training compute is therefore:

C = 6N x D   (D = training tokens)

A refinement matters at long context: plain 6N ignores the attention matmuls, which cost roughly 12LHQT extra FLOPs per token (L layers, H heads, Q head dimension, T sequence length) because they grow with sequence length while the weight matrices do not. PaLM reported the difference on its own model: 45.7% MFU counting weights only, 46.2% counting self-attention too. For a 405B model at 8K context the weights dominate; at 128K context, attention stops being a footnote. The Llama 3 paper shows this directly: the same 16,384-GPU cluster runs at 41% MFU at 8K context and 38% at 131K context.

The denominator uses published peak FLOPs at the precision the training actually uses, typically BF16 on tensor cores. This is where casual efficiency claims get twisted: NVIDIA's H100 SXM spec lists 1,979 teraFLOPS of BF16 tensor-core throughput under the standard 2:4 structured-sparsity caveat, while dense throughput is half that, 989 teraFLOPS. Divide by the sparsity number and your utilization halves on paper. Llama 3's reported 430 TFLOPs per GPU against the dense 989 peak is exactly the 43% the paper claims.

Why the metric exists: GPU-Util tells you nothing

Before MFU, teams reported raw throughput or NVIDIA's "GPU utilization" from nvidia-smi, which is defined as the percentage of a sample period during which at least one kernel was executing on the device. A single trivial kernel that never stops reports 100% GPU-Util. A data-loader-bound job where the chip computes almost nothing still shows "busy." You can hit 100% utilization on memory movement alone.

MFU fixes the comparison problem by normalizing across chips and cluster sizes. GPT-3 at 21.3% MFU and PaLM at 46.2% mean: for the same model math per token, PaLM's 6,144 TPU v4 chips converted more than twice as much theoretical peak into useful work as GPT-3's V100 fleet. That is a claim about software and interconnect, not model quality.

PaLM also introduced the paired metric HFU (Hardware FLOPs Utilization), which counts every FLOP the chip really executes, including activation rematerialization (recomputing forward-pass activations instead of storing them). PaLM 540B: 46.2% MFU, 57.8% HFU. The chips were busier than the model needed. When you see both numbers in a paper, HFU minus MFU is the recomputation tax, and MFU is the honest one, because the denominator is fixed by architecture alone.

What good numbers look like

  • 35 to 45%: the working range for well-tuned LLM pre-training runs, and what frontier labs report for trillion-token runs on mature stacks. Llama 3's 405B: 43% on 8,192 GPUs, dropping to 41% on 16,384 and 38% in the long-context stage.
  • Above 50%: excellent. PaLM's team attributed its then-record number partly to a training recipe that avoids pipeline parallelism entirely across two TPU v4 pods.
  • 20 to 30%: typical for older runs or badly tuned ones. GPT-3's 21.3% was not incompetence; it reflected the tooling of 2020.
  • A few percent: normal for LLM inference at small batch sizes. Decode is memory-bound, not compute-bound, so MFU is the wrong lens there. Databricks (MosaicML) proposed MBU, Model Bandwidth Utilization, as the inference-side counterpart: achieved memory bandwidth divided by peak. Their worked example: a 7B model in 16-bit precision generating one token every 14 milliseconds moves 14GB of weights in 14ms, or 1TB/s, which on a 2TB/s GPU is 50% MBU even though the compute units are nearly asleep.

The economics, with Llama 3's own numbers

The Llama 3 paper gives total pre-training compute as roughly 3.8 x 10^25 FLOPs for the 405B model. Run that through the MFU definition at Meta's reported numbers (16,384 H100s at 989 dense BF16 teraFLOPS each, 41% utilization) and you get about 66 days of wall clock, which matches Meta's description of a training run of several months on over 16,000 GPUs. The same run at a hypothetical 100% MFU would take 27 days, and the gap between them is the difference between about 10.7 million and 26 million H100-hours. At the floor of 2026 rental listings, roughly $1.40 per H100-hour, that is about $15 million versus $36 million. At this scale a single point of MFU is worth more than a day and a half of a 16,000-GPU cluster.

Where MFU leaks

Knowing the failure modes is how a platform team reads a paper, or finds money in its own cluster.

Communication and pipelining. Llama 3's 405B used 8-way tensor parallelism, 16-way pipeline parallelism, and up to 128 data-parallel replicas across 16,384 H100s. Pipeline stages wait on each other, tensor parallelism adds per-layer all-reduces, and data parallelism adds gradient sync. The paper's own numbers show the scale tax: from 43% MFU on 8,192 GPUs to 41% on 16,384, because the global batch stayed fixed while each replica's share of it halved.

Long context. The quadratic attention term grows while the weights term does not. 41% to 38% going from 8K to 131K context.

Rematerialization. Real and useful, but it shows up in HFU, not MFU. Labs trade memory for recompute deliberately.

Crashes and recovery. The Llama 3 team logged 466 job interruptions during a 54-day pre-training window, 419 of them unexpected, and attributes about 78% of those to confirmed or suspected hardware issues, mostly GPUs and their HBM3 memory. Every crash costs a rollback to the last checkpoint. Since MFU is measured from realized throughput, not clock cycles, all of that downtime lands in the number.

A balance beam tilted over two factory scenes: an orderly conveyor of blocks versus machines shuttling blocks back and forth

The caveats

MFU is not comparable across papers unless you check the fine print. It is precision-specific: 40% in BF16 on H100 does not translate to FP8 or to another chip's peak, and nobody normalizes energy. Labs are not required to report it, and when they do, they report it for their best configurations, not the average over the whole run (Llama 3's own 38-43% range spans three different configurations). Mixture-of-Experts models add a further choice: count FLOPs over active parameters and the number flatters, count over total parameters and the same run looks worse.

It also says nothing about whether the model is good. A 45%-MFU run on a mediocre architecture out-earns a 25% run on a great one only on the electric bill, not on capability. MFU measures the plumbing. It became the industry's standard quality stamp for plumbing because, unlike GPU-Util, it cannot be satisfied by a GPU that is merely awake.

Sources

educationalExplainers