Every large language model trained today runs on a standing bet about numbers: that almost all of its arithmetic can happen at half, or a quarter, of the precision a normal computer uses, while a few carefully chosen parts of the training loop stay exact. That bet has a name, mixed-precision training, and it is the reason a 405-billion-parameter model can be trained in months rather than decades. This explainer covers what the bet is, where it came from, and where it has gone.
What a floating point number trades away
A computer approximates real numbers with floating point formats. The standard 32-bit format, FP32, splits its bits into three fields: one sign bit, an 8-bit exponent that sets how large or small a number can be, and a 23-bit mantissa that sets how many digits of precision each number carries. The same number of bits can be split differently. IEEE half precision, FP16, also uses 16 bits in total but allocates only 5 bits to the exponent and 10 to the mantissa. The consequence is severe: FP16 cannot represent any finite value beyond about 65,504, while FP32 reaches roughly 3.4e38 (Google Cloud).
Neural network training turns out to tolerate the precision loss. Weights, activations and gradients do not need seven significant decimal digits to be useful. What they do need is range. A value that overflows to infinity, or underflows to zero, corrupts the math in a way a slightly rounded value does not. Almost every decision in mixed-precision training follows from this distinction between range and precision.

The 2017 recipe: three fixes for three losses
The modern technique comes from a 2017 paper by NVIDIA researchers (Micikevicius, Narang and colleagues). Their proposal: store weights, activations and gradients in FP16 during forward and backward passes, cutting memory use by nearly 2x and setting up large speedups on the half-precision hardware units then arriving in datacenter GPUs. But FP16 loses information in three places, and they supplied a fix for each.
First, weight updates are often tiny relative to the weight itself. Adding a small gradient to a large weight in 16-bit arithmetic just rounds the update away. The fix was to keep a full-precision FP32 copy of every weight, the master weights, and apply optimizer updates only to that copy, rounding it down to FP16 for the next pass. Second, many gradients are smaller than FP16's smallest usable values and flush to zero. An NVIDIA tutorial on the technique puts it bluntly: FP16 spans about 40 powers of two, but typical gradient distributions occupy only about 15 of them, leaving small values unrepresentable. The fix was loss scaling: multiply the loss by a large constant before backpropagation so the chain rule lifts every gradient into FP16's range, then divide the gradients back down before the FP32 update. Third, numerically delicate operations such as softmax and the reductions inside normalization layers can amplify rounding errors. Those stay in FP32.
All three pieces survive in every current recipe. The master-weights-plus-loss-scaling pattern is now handled automatically by libraries: PyTorch calls it automatic mixed precision, and framework documentation still describes training in the same terms the 2017 paper used.
Why bfloat16, not FP16, became the default
FP16 works, but it demands loss scaling and constant vigilance against its 65,504 ceiling. Google took a different route. The bfloat16 format ("brain float") was developed by Google Brain for the TPU v2 chip in 2017 and works because of one design choice: it keeps FP32's full 8-bit exponent and spends its 16 bits on just 7 explicit mantissa bits (Google Cloud blog). Its dynamic range is identical to FP32. It trades digits of precision, not reach. Converting an FP32 number to bfloat16 is nearly free, and overflow stops being a routine hazard.
A 2019 study from Intel and Facebook AI Research (Kalamkar et al.) tested BFLOAT16 across image classification, speech recognition, language modeling and recommendation systems and found it matched FP32 quality with no hyperparameter changes and no loss scaling needed. Once NVIDIA's A100 added BF16 to its tensor core type list in 2020, the format became the default for large-scale language model training. Meta's Llama 3 paper confirms the choice in its own way: with model parameters in BF16, the 405B model does not fit in the GPU memory of a single 8-H100 machine, which is part of why training it took a cluster of 16,000 H100 GPUs.
The A100 whitepaper also introduced an oddity worth knowing: TF32, a 19-bit internal format (8-bit exponent, 10-bit mantissa) that silently accelerates FP32 matrix multiplies inside tensor cores while programs still declare FP32 inputs and outputs. Nothing to rewrite, most of the speedup to collect.
What still never shrinks
The pattern across nine years of recipes is that certain quantities keep FP32 regardless of what the matmuls run in. Master weights and Adam's optimizer moments stay in FP32, because both accumulate very small changes over millions of steps. NVIDIA's own Llama 3 pretraining examples keep model storage and checkpoints in BF16 but hold FP32 master weights and optimizer state, and the docs note that the two patterns "converge equivalently" precisely because that FP32 bookkeeping layer is preserved. The precision ladder is not a march toward smaller numbers everywhere. It is a redistribution: fast and small where the math is forgiving, slow and exact at the accumulation points.

The 8-bit step, and its price
Hopper GPUs (2022) added native FP8 compute, backed by a joint specification from NVIDIA, Arm and Intel that defines two 8-bit encodings. E4M3 (4 exponent bits, 3 mantissa bits) reaches only 448 but carries more precision; E5M2 reaches 57,344 with two mantissa bits. The standard usage, documented in NVIDIA's Transformer Engine primer, runs the forward pass in E4M3 where precision matters and the backward pass in E5M2 where gradients need range, with per-tensor scaling factors chosen from observed maxima.
The first frontier-scale demonstration came from DeepSeek. The DeepSeek-V3 technical report (December 2024) trained its 671B-parameter mixture-of-experts model end-to-end in FP8 for the bulk of its matrix multiplies, at a total of 2.788 million H800 GPU hours, a format choice the report credits with roughly double the theoretical matrix-multiply throughput versus BF16 on that hardware. It also shows what the step costs. DeepSeek found that H800 FP8 accumulation retains only about 14 bits of precision, far below FP32, which produces visible errors in large-inner-dimension multiplies; the team fixed it by copying partial sums into FP32 registers every 128 elements. After that and a list of other audits, its controlled comparison against BF16 baselines kept the loss-curve relative error below 0.25%. FP8 training works, at scale, when someone has the discipline to check every accumulation path.
The 4-bit frontier
Two more steps are visible now, both for inference-first reasons. The Open Compute Project's Microscaling standard (2023) defines block-scaled formats like MXFP4: 4-bit floating point elements grouped with a shared scale factor per block. NVIDIA's Blackwell generation implements a close variant, NVFP4, with 16-element groups and FP8 scale factors, at roughly 4.5 bits per element on average.
On the training side, Moonshot's Kimi K2 Thinking (November 2025) went a different way: a 1-trillion-parameter MoE released natively in INT4, with quantization-aware training applied to the MoE components during post-training, yielding roughly 2x faster generation and a checkpoint of about 594GB instead of over 1TB, according to Artificial Analysis. The distinction from ordinary quantization is intent: post-training quantization compresses a model that was trained assuming full precision and hopes it survives; QAT trains the model to survive from the start.
Why labs keep going down anyway
Each rung of the ladder roughly halves the bytes per number and doubles the hardware's matrix throughput, per the same progression from the 2017 paper through the A100 and H100 generations. Halved bytes are not only faster math: they are cheaper memory, cheaper inter-GPU communication, and bigger batches per device, which is where DeepSeek explicitly points the finger when it says FP8 storage and communication let it train without costly tensor parallelism. The floor is real but soft. Accumulation paths, optimizer state and normalization reductions keep demanding higher precision, and each format step so far has been paid for with engineering audit rather than free.
Key takeaways
- Mixed-precision training means running cheap 16-, 8- or 4-bit arithmetic in the bulk of the forward and backward passes while keeping FP32 master weights, optimizer moments and sensitive reductions exact. The division of labor dates to NVIDIA's 2017 paper and has never changed.
- The important bit in these formats is the exponent. BF16 won the 2020s because it kept FP32's range and sacrificed only mantissa digits, removing the need for loss scaling.
- FP8 at frontier scale (DeepSeek-V3) required discovering and patching a hardware precision flaw, 14-bit accumulation on H800, before the loss curves matched BF16.
- The 4-bit era (MXFP4/NVFP4 on Blackwell, Kimi K2 Thinking's INT4 QAT) blurs training-time and inference-time precision into one design problem: train the model for the format it will serve in.
Sources
- Mixed Precision Training, Micikevicius, Narang et al., 2017
- Introduction to Mixed Precision Training (tutorial), NVIDIA Research, ICCV 2019
- Bfloat16: The secret to high performance on Cloud TPUs, Google Cloud, 2019
- Bfloat16 floating-point format, Wikipedia
- A Study of BFLOAT16 for Deep Learning Training, Kalamkar et al., Intel and Facebook AI Research, 2019
- NVIDIA A100 Tensor Core GPU Architecture whitepaper, NVIDIA, 2020
- The Llama 3 Herd of Models, Meta, 2024
- FP8 Formats for Deep Learning, Micikevicius et al. (NVIDIA, Arm, Intel), 2022
- Using FP8 and FP4 with Transformer Engine, NVIDIA documentation
- DeepSeek-V3 Technical Report, DeepSeek, 2024
- Mixed-Precision Training, NeMo AutoModel docs, NVIDIA
- Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization, 2025
- Kimi-K2-Thinking model card, Moonshot AI, 2025
- Kimi K2-Thinking, everything you need to know, Artificial Analysis, 2025
