FP8 Mixed-Precision Training: Formats, Scaling Recipes, and Fine-Grained GEMMs

Training frontier large language models requires navigating harsh memory bandwidth and compute boundaries. For years, 16-bit formats such as IEEE FP16 and Brain Floating Point (BF16) served as the standard arithmetic precision for deep learning workloads. While BF16 provided sufficient dynamic range to prevent gradient underflow without manual loss scaling, training clusters still spent substantial time bounded by memory capacity and interconnect bandwidth.

The transition to 8-bit floating-point (FP8) precision addresses these scaling constraints directly. FP8 halves tensor memory footprints, doubles peak arithmetic throughput on modern Tensor Cores, and reduces inter-node collective communication volume. However, reducing numerical precision to 8 bits introduces acute quantization challenges: representational collapse, severe underflow in gradient computation, and catastrophic error propagation caused by activation outliers.

Stabilizing FP8 throughout pretraining requires specialized numerical formats, dynamic scaling strategies, fine-grained tile-wise quantization, and high-precision accumulation in hardware execution units.

The Arithmetic Bottlenecks of 16-Bit Training

Modern autoregressive transformer architectures scale compute costs with parameter count N and token count D, following roughly 6ND floating-point operations (FLOPs) per training step across forward and backward passes. In standard 16-bit mixed precision (using BF16 or FP16), memory demands decompose into several components:

• Model Parameters and Optimizer States: AdamW maintains master weights in FP32 (4 bytes per parameter), first-moment momentum (4 bytes per parameter), second-moment variance (4 bytes per parameter), and BF16 model weights (2 bytes per parameter), totaling 16 to 18 bytes per parameter before sharding.
• Activation Footprint: Storing intermediate activations for the backward pass scales linearly with sequence length, batch size, hidden dimensions, and attention layer depth.
• Communication Overhead: Distributed training schemes, including Fully Sharded Data Parallel (FSDP), Tensor Parallelism (TP), and Expert Parallelism (EP), require continuous high-bandwidth all-gather, reduce-scatter, and all-to-all communications across GPUs.

Operating Tensor Cores at 16 bits leaves significant compute throughput untapped. Hardware architectures like NVIDIA Hopper and Blackwell provide double the theoretical dense arithmetic FLOPs for 8-bit operations compared to 16-bit operations (such as 1,979 TFLOPS of FP8 dense compute on an NVIDIA H100 SXM5 versus 989 TFLOPS of BF16 compute). Halving the byte width of weights, activations, and gradients doubles arithmetic density while halving memory bandwidth demand.

Bit Allocation: E4M3 vs. E5M2 Mechanics

Standard floating-point representations allocate bits across three fields: a sign bit (S), exponent bits (E), and mantissa (fraction) bits (M). The value of a normal floating-point number is computed as:

Value = (-1)^S * 2^(E - bias) * (1 + M / 2^mantissa_bits)

In an 8-bit budget, distributing bits between exponent and mantissa forces a strict trade-off between dynamic range and precision. The Open Compute Project (OCP) and hardware vendors standardized two distinct FP8 formats for deep learning workloads, documented by Micikevicius et al. (2022):

FP8 Format Specifications:

1. E4M3 (High Precision, Narrow Dynamic Range):
   - Bit Layout: 1 sign bit, 4 exponent bits, 3 mantissa bits
   - Exponent Bias: 7
   - Max Representable Absolute Value: 448
   - Minimum Positive Normal Value: ~0.015625 (2^-6)
   - Special Value Handling: S.1111.111 reserved for NaN (no infinity representation)

2. E5M2 (Wide Dynamic Range, Lower Precision):
   - Bit Layout: 1 sign bit, 5 exponent bits, 2 mantissa bits
   - Exponent Bias: 15
   - Max Representable Absolute Value: 57,344
   - Minimum Positive Normal Value: ~6.10e-5 (2^-14)
   - Special Value Handling: Standard IEEE representation (supports +Inf, -Inf, and NaN)

Comparison with 16-bit baselines:
- BF16: 1 sign, 8 exponent, 7 mantissa (Max: ~3.39e38, Min: ~1.17e-38)
- FP16: 1 sign, 5 exponent, 10 mantissa (Max: 65,504, Min: ~6.10e-5)

E4M3: Precision for Forward Tensors

E4M3 dedicates 3 bits to the mantissa, yielding 8 discrete quantization levels between adjacent powers of two. To maximize the narrow representational range, E4M3 drops support for infinities; overflow saturates directly or is designated as NaN. The maximum absolute representable value is 448. Because forward activations and weights require high precision to avoid degradation in model perplexity, E4M3 is primarily deployed for forward-pass general matrix multiplications (GEMMs).

E5M2: Dynamic Range for Backward Gradients

E5M2 mirrors the exponent structure of IEEE FP16 (5 bits, bias 15), providing an identical dynamic range extending up to 57,344 and down to subnormal values near 1.52e-5. However, with only 2 mantissa bits, it provides only 4 discrete levels per bin. Gradients in deep networks typically span several orders of magnitude, making dynamic range essential to prevent gradient underflow. Consequently, E5M2 is suited for backward gradient tensors where precision loss is tolerable but underflow truncates learning signals.

Hybrid FP8 Formats and Scaling Recipes

Naive conversion of 16-bit tensors into FP8 leads to immediate instability because values routinely exceed 448 (causing saturation) or drop below 0.015 (flushing to zero). To utilize the full dynamic range of FP8, every tensor must be multiplied by a scaling factor S before quantization:

X_fp8 = clip(round(X * S), -MAX_FP8, MAX_FP8)

During matrix multiplication, the inverse scale factors dequantize the output back to higher precision:

Y = (A_fp8 * B_fp8) * (1 / (S_A * S_B))

Hybrid FP8 Execution Flow:

Forward Pass:
- Input Activation X (E4M3, scaled by Sx) * Weight W (E4M3, scaled by Sw)
  --> Accumulated in FP32 Tensor Cores
  --> Dequantized output Y in BF16

Backward Activation Gradients:
- Gradient dY (E5M2, scaled by S_dy) * Weight W^T (E4M3, scaled by Sw)
  --> Output dX in BF16 for backpropagation

Backward Weight Gradients:
- Gradient dY^T (E5M2, scaled by S_dy) * Input X (E4M3, scaled by Sx)
  --> Output dW in FP32/BF16 for AdamW optimizer updates

The Delayed Scaling Algorithm

Calculating the optimal scaling factor S = MAX_FP8 / max(|X|) dynamically before each GEMM requires an extra reduction pass over the tensor. On distributed clusters, global maximum reductions introduce kernel launch latencies and synchronization stalls that erode compute speedups.

To bypass this bottleneck, NVIDIA Transformer Engine introduced the Delayed Scaling algorithm. Instead of calculating the maximum absolute value (amax) of the current tensor synchronously, the system tracks a rolling history window of amax values across the preceding N iterations (typically N = 16 to 1024).

The scaling factor for step t is calculated from the historical maximum:

• amax_est = max(amax[t-1], amax[t-2], ..., amax[t-N])
• S[t] = MAX_FP8 / amax_est

If the current tensor value unexpectedly surges past amax_est, the tensor values saturate at MAX_FP8. Because neural network activations change smoothly across adjacent gradient steps under small learning rates, delayed scaling achieves near-optimal scaling without extra synchronization passes.

Outliers and Fine-Grained Block Quantization

While delayed per-tensor scaling functions reliably for dense models under 10 billion parameters, it encounters severe limitations in larger language models and Mixture-of-Experts (MoE) architectures, as analyzed in Peng et al. (2023).

In large models, individual feature dimensions systematically develop extreme activation outliers, values up to 100 times larger than the median activation magnitude. When a single outlier dictates the tensor-wide maximum amax, the resulting scaling factor S is forced to a very small number. Consequently, the remaining 99.9% of normal activations are compressed into the smallest FP8 bins or flushed to zero, leading to catastrophic representation collapse.

Per-Tensor Quantization (Flawed at scale):
[ 0.02,  0.05,  0.01,  184.0 (Outlier),  0.03,  0.04 ]  --> Scaled by 448/184 = 2.43
Quantized: [  0,     0,     0,     448,     0,     0  ]  --> 80%+ information destroyed

Fine-Grained Tile-Wise Quantization (DeepSeek-V3 / MXFP8):
Tile 0: [ 0.02, 0.05, 0.01 ] --> S0 = 448 / 0.05 = 8960  --> [ 179, 448, 90 ]
Tile 1: [ 184.0, 0.03, 0.04 ] --> S1 = 448 / 184  = 2.43  --> [ 448,   0,  0 ]
Local representation preserved across unaffected tiles.

Tile-Wise and Block-Wise Quantization in DeepSeek-V3

To resolve outlier degradation during full-scale pretraining of a 671B parameter model, DeepSeek-AI (2024) implemented a fine-grained FP8 quantization framework:

• Activation Quantization (Tile-Wise): Activations are partitioned into 1x128 tiles (1 token across 128 channel elements). Each 128-element group receives an independent scaling factor computed in registers, isolating channel outliers to their specific sub-vector.
• Weight Quantization (Block-Wise): Model parameter matrices are partitioned into 2D blocks of 128x128 elements. Each 128x128 block maintains its own independent FP8 scaling factor, preserving weight resolution across distinct projection heads.
• Backward Transposition: Because tile-wise quantization requires 1x128 grouping in the forward pass and 128x1 grouping for activation gradients in the backward pass, weights are kept in 128x128 blocks so that standard matrix transposition yields identical block layouts without data restructuring.

The OCP Microscaling (MX) Standard

This approach aligns with the Open Compute Project Microscaling Formats (MX) specification developed by AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm (Rouhani et al., 2023). The MXFP8 specification groups 32 consecutive elements into a microscaling block sharing an 8-bit E8M0 scale factor. The hardware reads the shared scale and individual 8-bit elements, performing fast dot-products inside execution pipelines without full per-tensor dequantization in global memory.

Hardware Execution: Tensor Core Accumulation

A common misconception is that FP8 matrix multiplication computes all internal additions in 8-bit precision. In practice, summing thousands of 8-bit floating-point products in FP8 causes catastrophic precision loss through numerical cancellation and roundoff swamping (adding small products to large partial sums).

In modern hardware architectures (such as NVIDIA 4th-generation Tensor Cores in Hopper and 5th-generation in Blackwell), the arithmetic pipeline executes as follows:

1. Input Loading: Matrix tiles A and B are loaded from shared memory into register files in FP8 (E4M3 or E5M2).
2. Multiplier Array: The internal fused multiply-add (FMA) units compute dot-products of FP8 elements, expanding the products to higher precision (FP16 or FP32).
3. Accumulator Registers: The partial sums are accumulated in FP32 (or BF16 intermediate accumulators). For an inner dimension K = 8192, the accumulator sums 8,192 products in single-precision floating point.
4. Epilogue: Activation functions (such as SwiGLU or GELU), bias additions, and output casting are performed in FP32 or BF16 before writing the output tensor back to High Bandwidth Memory (HBM).

Custom GEMM libraries like DeepGEMM optimize this pipeline by keeping scaling factor multiplications inside the Tensor Core accumulator epilogue, avoiding extra memory trips between Tensor Cores and CUDA cores.

Production Impact: Memory, Compute, and Cluster Communication

Deploying fine-grained FP8 across modern pretraining pipelines delivers performance gains across several key operational dimensions:

• Compute Speedups: Because FP8 matrix multiplication units require half the silicon area and power of 16-bit multipliers, chip designers pack twice as many FP8 ALUs into each Streaming Multiprocessor. Pretraining GEMM-heavy layers (such as dense MLPs and MoE routed experts) achieves up to a 1.7x to 1.9x real-world throughput increase over BF16 baselines.
• Activation Cache Halving: In long-context training, intermediate activation storage frequently exhausts GPU HBM, forcing teams to use heavy activation checkpointing (gradient recomputation). Quantizing saved activations to FP8 halves the activation memory footprint, allowing larger per-GPU micro-batch sizes and reducing activation recomputation overhead.
• MoE Communication Compression: In large Mixture-of-Experts models distributed via Expert Parallelism (EP), GPUs must route tokens to remote expert nodes via an all-to-all communication primitive. Transporting token representations in FP8 rather than BF16 reduces communication payload volumes by 50%, mitigating interconnect saturation across InfiniBand and NVLink fabrics.
• Master Parameter Integrity: Master weights and optimizer state accumulation remain in full FP32/BF16 inside the AdamW optimizer step, ensuring that subtle weight updates are not lost to quantization noise.

Summary of the FP8 Training Stack

FP8 Production Component Mapping:

- Model Weights (Forward):      FP8 (E4M3) with 128x128 Block-Wise Scaling
- Activations (Forward):        FP8 (E4M3) with 1x128 Tile-Wise Scaling
- Gradients (Backward):         FP8 (E5M2 or E4M3) with Dynamic Scaling
- Tensor Core Accumulation:     FP32 (Full Single-Precision Summation)
- Master Parameters:            FP32 (Maintained in AdamW Optimizer)
- Optimizer States (m, v):      FP32 / BF16 (High-Precision Accumulators)
- Softmax and Normalizations:   FP32 / BF16 (Kept in 16/32-bit to Prevent Underflow)

FP8 mixed-precision training transitions deep learning from uniform 16-bit computation to an optimized numerical pipeline. By pairing E4M3 and E5M2 bit structures with fine-grained block scaling and high-precision accumulator hardware, modern pretraining frameworks achieve significant compute and memory efficiency while preserving baseline numerical convergence.

Sources

• Micikevicius, P., Stosic, D., Judd, P., et al. (2022). FP8 Formats for Deep Learning. arXiv:2209.05433
• DeepSeek-AI. (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437
• Peng, H., Wu, K., Wei, Y., et al. (2023). FP8-LM: Training FP8 Large Language Models. arXiv:2310.18313
• Rouhani, B. D., et al. (2023). Microscaling Data Formats for Deep Learning. arXiv:2310.10537
• NVIDIA Corporation. (2023). Using FP8 with Transformer Engine. NVIDIA Documentation