Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training

In deep transformer architectures, weight initialization is the primary determinant of whether a trillion-token pre-training run converges smoothly or diverges during the first thousand steps. When training networks with 80 to 120 layers (such as Llama 3 70B, GPT-4, or deep mixture-of-experts models), naive application of classical Gaussian or uniform initializations causes catastrophic signal degradation. Activation magnitudes along the residual stream either explode exponentially or saturate normalization layers, while backward gradients vanish or trigger loss spikes.

Scaling transformers to massive depths requires precise control over signal propagation and variance accumulation across hundreds of additive residual connections. Modern frontier models combine variance-scaled projections, specialized residual multipliers, and bounded normalization formulations such as DeepNorm to ensure stable gradient dynamics from step zero.

The Residual Variance Accumulation Problem

The transformer architecture relies fundamentally on residual skip connections. For any transformer layer $l$, the forward propagation through an attention or multi-layer perceptron (MLP) sublayer $F_l$ is expressed as:

$$x_{l+1} = x_l + F_l(x_l)$$

Here, $x_l \in \mathbb{R}^{B \times S \times d}$ represents the hidden state tensor on the residual stream, where $B$ is batch size, $S$ is sequence length, and $d$ is the hidden dimension.

Linear Variance Growth with Depth

Assume that at initialization, the input embedding $x_0$ has zero mean and unit variance ($\mathbb{E}[x_0] = 0$, $\text{Var}(x_0) = 1$). Assume further that each sublayer function $F_l$ produces activations with zero mean and variance $\sigma^2_F$, independent of $x_l$.

Because variances add linearly across independent random variables:

$$\text{Var}(x_l) = \text{Var}(x_0) + \sum_{i=0}^{l-1} \text{Var}(F_i(x_i)) = 1 + l \cdot \sigma^2_F$$

In an unscaled $L$-layer transformer with $2L$ sublayers (one self-attention block and one MLP block per layer), the activation variance at the final layer scales as:

$$\text{Var}(x_{2L}) = 1 + 2L \cdot \sigma^2_F \approx O(L)$$

For an 80-layer model ($2L = 160$ sublayers), the signal magnitude on the residual stream grows by more than an order of magnitude from the bottom layer to the top layer.

Layer 0 (Embeddings):    Var(x_0)   = 1.00
Layer 20 (Sublayer 40):  Var(x_40)  = 1.00 + 40 * σ²_F  (~21.0 if σ²_F = 0.5)
Layer 40 (Sublayer 80):  Var(x_80)  = 1.00 + 80 * σ²_F  (~41.0)
Layer 80 (Sublayer 160): Var(x_160) = 1.00 + 160 * σ²_F (~81.0)

Why Classical Initialization Schemes Fail

Classical initialization methods were designed for feedforward and convolutional networks without deep additive identity paths:

1. Xavier / Glorot Initialization (Glorot & Bengio, 2010):

Sets weight variance to: $$\text{Var}(W) = \frac{2}{n_{\text{in}} + n_{\text{out}}}$$ This ensures $\text{Var}(y) = \text{Var}(x)$ across an isolated matrix multiplication $y = Wx$. However, it ignores the additive accumulation of the residual connection $x + Wx$, causing $\text{Var}(x_l)$ to compound across layers.

1. He / Kaiming Initialization (He et al., 2015):

Sets weight variance to: $$\text{Var}(W) = \frac{2}{n_{\text{in}}}$$ Designed to account for ReLU non-linearities, Kaiming initialization also assumes sequential transformation without residual summation, accelerating variance explosion in deep transformers.

Residual Projection Scaling: The $1/\sqrt{2L}$ Rule

To counteract linear variance accumulation, OpenAI introduced residual projection scaling in GPT-2 (Radford et al., 2019), which was later formalized across large-scale distributed training by Megatron-LM (Shoeybi et al., 2019).

The Scaling Mechanism

In standard transformer layers, weights are initialized from a normal distribution $\mathcal{N}(0, \sigma^2)$, where $\sigma = \sqrt{2 / (5 d)}$ or $\sigma = 0.02$.

For the output projection matrices that write directly back into the residual stream:

• The attention output projection matrix: $W_o \in \mathbb{R}^{d \times d}$
• The MLP down-projection matrix: $W_{\text{down}} \in \mathbb{R}^{d_{\text{ffn}} \times d}$

Their initial weights are scaled down by a factor of $\frac{1}{\sqrt{2L}}$, where $L$ is the total number of transformer layers:

$$W_o \sim \mathcal{N}\left(0, \frac{\sigma^2}{2L}\right), \quad W_{\text{down}} \sim \mathcal{N}\left(0, \frac{\sigma^2}{2L}\right)$$

Mathematical Proof of Variance Stability

When output projections are scaled by $\frac{1}{\sqrt{2L}}$, the variance contributed by each sublayer is reduced to $\text{Var}(F_l(x_l)) = \frac{\sigma^2_F}{2L}$.

The variance at the final layer $2L$ becomes:

$$\text{Var}(x_{2L}) = \text{Var}(x_0) + \sum_{l=0}^{2L-1} \frac{\sigma^2_F}{2L} = 1 + 2L \cdot \left(\frac{\sigma^2_F}{2L}\right) = 1 + \sigma^2_F = O(1)$$

By dampening the initial contribution of every residual branch, the model maintains a constant $O(1)$ activation variance from input to output at initialization.

Pre-LN vs. Post-LN Initialization Dynamics

The location of layer normalization fundamentally changes how initialization errors propagate through the model during forward and backward passes (Xiong et al., 2020).

Post-LN: Vanishing Gradients and the Warmup Dependency

In the original Post-LN transformer (Vaswani et al., 2017):

$$x_{l+1} = \text{LayerNorm}(x_l + F_l(x_l))$$

Because normalization is applied to the sum, the gradient with respect to layer $l$ is scaled by the denominator of the LayerNorm operation:

$$\frac{\partial \mathcal{L}}{\partial x_l} \approx \frac{\partial \mathcal{L}}{\partial x_{l+1}} \cdot \frac{1}{\sqrt{\text{Var}(x_l + F_l(x_l)) + \epsilon}}$$

As depth increases, activations near the top layers exhibit large variance prior to normalization, forcing LayerNorm to divide by large scale factors. This causes gradients flowing to early layers to vanish exponentially unless an aggressive learning rate warmup schedule is applied over tens of thousands of steps.

Pre-LN: Gradient Stability at the Expense of Effective Depth

To eliminate fragile learning rate warmups, modern architectures adopted Pre-LN (Radford et al., 2019):

$$x_{l+1} = x_l + F_l(\text{LayerNorm}(x_l))$$

Because normalization is applied only inside the sublayer branch, gradients flow directly through the identity connection without attenuation:

$$\frac{\partial \mathcal{L}}{\partial x_0} = \frac{\partial \mathcal{L}}{\partial x_L} + \sum_{l=0}^{L-1} \frac{\partial F_l}{\partial x_l}$$

However, Pre-LN introduces a subtle failure mode: the residual stream norm $\|x_l\|$ grows as $\sqrt{l}$. Since LayerNorm normalizes $x_l$ to unit scale before passing it to $F_l$, the relative update contributed by deeper sublayers diminishes:

$$\frac{\|F_l(\text{LN}(x_l))\|}{\|x_l\|} \approx \frac{1}{\sqrt{l}}$$

In very deep Pre-LN transformers (e.g., beyond 50 layers), upper layers contribute negligible updates relative to the accumulated residual stream, effectively capping the model's expressive capacity.

DeepNorm: Scaling Transformers to 1,000 Layers

To resolve the trade-offs of Pre-LN and Post-LN, researchers at Microsoft introduced DeepNorm (Wang et al., 2022). DeepNorm modifies both the residual connection scaling and the initialization variance of sublayer weights to bound the expected change in model parameters during optimization.

       Residual Path (x_l) ───[ * α ]───(+)───[ LayerNorm ]───> x_{l+1}
                                         │
       Sublayer Path ─────────[ F_l ]────┘
                              (weights initialized with β)

DeepNorm Architecture Formulation

DeepNorm modifies the residual addition by introducing a constant scaling factor $\alpha$ on the identity branch and an initialization scale factor $\beta$ on the sublayer weights:

$$x_{l+1} = \text{LayerNorm}\left(x_l \cdot \alpha + F_l(x_l, \theta_l \cdot \beta)\right)$$

Theoretical Derivation of $\alpha$ and $\beta$

DeepNorm establishes a theoretical bound ensuring that the model update magnitude $\|\Delta F_l\|$ does not grow with network depth. For a transformer with $N$ layers (or $M$ decoder layers):

1. Decoder-Only or Encoder-Only Architecture ($L$ layers):

$$\alpha = (2L)^{1/4}$$ $$\beta = (8L)^{-1/4}$$

1. Encoder-Decoder Architecture ($N$ encoder layers, $M$ decoder layers):

• Encoder sublayers: $\alpha_{\text{enc}} = (2N)^{1/4}, \quad \beta_{\text{enc}} = (8N)^{-1/4}$
• Decoder sublayers: $\alpha_{\text{dec}} = (2M)^{1/4}, \quad \beta_{\text{dec}} = (8M)^{-1/4}$

Initialization Scale Table for DeepNorm

| Model Depth ($L$) | Identity Multiplier ($\alpha$) | Weight Scale Factor ($\beta$) | $1/\sqrt{2L}$ Megatron Baseline | | :--- | :--- | :--- | :--- | | 12 Layers (Base) | $2.213$ | $0.319$ | $0.204$ | | 32 Layers (7B) | $2.828$ | $0.250$ | $0.125$ | | 80 Layers (70B) | $3.557$ | $0.199$ | $0.079$ | | 200 Layers | $4.472$ | $0.158$ | $0.050$ | | 1,000 Layers | $6.687$ | $0.106$ | $0.022$ |

By scaling the identity path by $\alpha > 1$ and suppressing initial sublayer weights by $\beta < 1$, DeepNorm bounds activation growth inside the LayerNorm function, enabling stable training of 1,000-layer transformers without learning rate warmup or gradient divergence.

Zero-Initialization and Dynamic Multipliers: ReZero and Fixup

Alternative approaches eliminate layer normalization altogether or initialize deep networks as exact mathematical identities.

ReZero: Residual Zero-Initialization

ReZero (Bachlechner et al., 2020) introduces a single learnable scalar parameter $\alpha_l$ for each sublayer, initialized to zero:

$$x_{l+1} = x_l + \alpha_l F_l(x_l), \quad \text{where } \alpha_l = 0 \text{ at } t=0$$

At step zero, every sublayer output is multiplied by 0:

$$x_L = x_{L-1} = \dots = x_0$$

The entire deep transformer functions as an exact identity mapping at step zero:

• Signal propagates through 100+ layers with zero attenuation or explosion.
• Gradients $\frac{\partial \mathcal{L}}{\partial x_0}$ equal $\frac{\partial \mathcal{L}}{\partial x_L}$ directly.
• As optimization proceeds, the network dynamically learns how much capacity to recruit from each sublayer by updating $\alpha_l$.

Fixup: Fixed-Update Initialization

Fixup (Zhang et al., 2019) eliminates normalization layers by scaling weight initialization matrices to guarantee that the variance of the change in activations per layer is $O(1/L^2)$, preventing gradient explosion without runtime normalization overhead.

Modern Frontier Pre-Training Initialization Recipes

Modern open-weight and proprietary models (such as Llama 3, Mistral, Gemma 2, and DeepSeek) follow standardized initialization protocols tailored to modern architectural components:

┌─────────────────────────────────────────────────────────────────────────┐
│              Standard Modern LLM Initialization Recipe                  │
├──────────────────────────┬──────────────────────────────┬───────────────┤
│ Component                │ Distribution                 │ Scale (Std)   │
├──────────────────────────┼──────────────────────────────┼───────────────┤
│ Token Embeddings         │ Truncated Normal             │ σ = 0.02      │
│ Attention Q, K, V        │ Truncated Normal             │ σ = √(2/d)    │
│ Attention Output (W_o)   │ Truncated Normal (Scaled)    │ σ = √(2/d)/√2L│
│ MLP Gate / Up Projections│ Truncated Normal             │ σ = √(2/d)    │
│ MLP Down Projection      │ Truncated Normal (Scaled)    │ σ = √(2/d)/√2L│
│ RMSNorm Weights          │ Constant                     │ 1.0           │
│ Biases (if present)      │ Constant                     │ 0.0           │
└──────────────────────────┴──────────────────────────────┴───────────────┘

1. Truncated Normal vs. Standard Gaussian

Modern pre-training harnesses draw weights from a truncated normal distribution rather than an unbounded Gaussian:

$$W \sim \text{TruncatedNormal}(\mu=0, \sigma^2, a=-3\sigma, b=3\sigma)$$

Truncation at $\pm 2\sigma$ or $\pm 3\sigma$ eliminates extreme outlier weights in large parameter matrices (which contain billions of elements), preventing localized activation spikes from saturating activation functions (such as SwiGLU) during early iterations.

2. Dimension-Aware Scaling ($\sqrt{2 / 5d}$ vs. $\sqrt{1 / d}$)

When models scale to massive hidden dimensions ($d_{\text{model}} = 8192$ for 70B models), using fixed $\sigma = 0.02$ causes matrix multiplication outputs to grow excessively before normalization. Modern implementations scale base standard deviation inversely with dimension:

$$\sigma_{\text{base}} = \sqrt{\frac{2}{5 \cdot d_{\text{model}}}} \quad \text{or} \quad \sigma_{\text{base}} = \frac{1}{\sqrt{d_{\text{model}}}}$$

Production PyTorch Implementation

Below is a complete, production-grade implementation of weight initialization for a modern transformer block incorporating truncated normal sampling, residual projection scaling ($1/\sqrt{2L}$), and RMSNorm initialization.

import math
import torch
import torch.nn as nn

def init_transformer_weights(
    module: nn.Module,
    num_layers: int,
    d_model: int,
    std_scale: float = 1.0,
    truncation_factor: float = 3.0,
):
    """
    Applies modern variance-scaled truncated normal initialization 
    to transformer layers, following Megatron-LM and LLaMA conventions.
    """
    # Base standard deviation scaled by hidden dimension
    base_std = std_scale * math.sqrt(2.0 / (5.0 * d_model))
    # Residual output projection standard deviation scaled by 1/sqrt(2L)
    residual_std = base_std / math.sqrt(2.0 * num_layers)

    def _init_weight(tensor: torch.Tensor, std: float):
        bound = truncation_factor * std
        nn.init.trunc_normal_(tensor, mean=0.0, std=std, a=-bound, b=bound)

    if isinstance(module, nn.Linear):
        # Check if the linear layer is a residual output projection
        if getattr(module, "_is_residual_projection", False):
            _init_weight(module.weight, residual_std)
        else:
            _init_weight(module.weight, base_std)
        
        if module.bias is not None:
            nn.init.zeros_(module.bias)

    elif isinstance(module, nn.Embedding):
        _init_weight(module.weight, base_std)

    elif isinstance(module, (nn.LayerNorm, nn.RMSNorm)):
        if hasattr(module, "weight") and module.weight is not None:
            nn.init.ones_(module.weight)
        if hasattr(module, "bias") and module.bias is not None:
            nn.init.zeros_(module.bias)


class ModernTransformerBlock(nn.Module):
    def __init__(self, d_model: int, num_heads: int, ffn_dim: int, num_layers: int):
        super().__init__()
        self.d_model = d_model
        self.num_layers = num_layers

        # Self-Attention sublayer
        self.q_proj = nn.Linear(d_model, d_model, bias=False)
        self.k_proj = nn.Linear(d_model, d_model, bias=False)
        self.v_proj = nn.Linear(d_model, d_model, bias=False)
        self.out_proj = nn.Linear(d_model, d_model, bias=False)
        self.out_proj._is_residual_projection = True  # Tag for residual scaling

        # MLP / Feed-Forward sublayer (SwiGLU style)
        self.gate_proj = nn.Linear(d_model, ffn_dim, bias=False)
        self.up_proj = nn.Linear(d_model, ffn_dim, bias=False)
        self.down_proj = nn.Linear(ffn_dim, d_model, bias=False)
        self.down_proj._is_residual_projection = True  # Tag for residual scaling

        # Layer Normalization
        self.input_norm = nn.RMSNorm(d_model)
        self.post_attention_norm = nn.RMSNorm(d_model)

        # Initialize all sub-modules
        self.apply(lambda m: init_transformer_weights(m, num_layers, d_model))

Architecture Comparison and Failure Modes

| Initialization Method | Residual Formula | Sublayer Initialization | Primary Advantage | Primary Failure Mode | | :--- | :--- | :--- | :--- | :--- | | Xavier / Glorot | $x_{l+1} = x_l + F_l(x_l)$ | $\sigma = \sqrt{\frac{2}{n_{\text{in}} + n_{\text{out}}}}$ | Standardizes isolated linear maps | Linear variance accumulation $O(L)$ in deep networks | | Megatron $1/\sqrt{2L}$ | $x_{l+1} = x_l + F_l(x_l)$ | Projections: $\sigma = \frac{\sigma_0}{\sqrt{2L}}$ | Preserves constant variance $O(1)$ at step 0 | Deeper layers contribute less relative signal in Pre-LN | | DeepNorm | $x_{l+1} = \text{LN}(x_l \cdot \alpha + F_l)$ | Sublayer: $W \sim \mathcal{N}(0, \beta^2)$ | Enables 1,000-layer scaling without warmup | Requires architectural parameter modification ($\alpha, \beta$) | | ReZero | $x_{l+1} = x_l + \alpha_l F_l(x_l)$ | Standard; $\alpha_l = 0$ | Network starts as exact identity | Slower early parameter learning if $\alpha_l$ updates slowly | | Fixup | Scaled residual branch | Scaled by $L^{-1/2}$ or $L^{-1/4}$ | Removes normalization layers entirely | Hyper-sensitive to learning rate schedule and depth |

Sources

• Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS 2010. https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf
• He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv:1502.01852. https://arxiv.org/abs/1502.01852
• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. https://arxiv.org/abs/1706.03762
• Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI Technical Report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
• Shoeybi, M., et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053. https://arxiv.org/abs/1909.08053
• Zhang, H., Dauphin, Y. N., & Ma, T. (2019). Fixup Initialization: Residual Learning Without Normalization. arXiv:1901.09321. https://arxiv.org/abs/1901.09321
• Xiong, R., et al. (2020). On Layer Normalization in the Transformer Architecture. ICML 2020. https://arxiv.org/abs/2002.04745
• Bachlechner, T., et al. (2020). ReZero is All You Need: Fast Convergence at Large Depth. arXiv:2003.04887. https://arxiv.org/abs/2003.04887
• Wang, H., et al. (2022). DeepNet: Scaling Transformers to 1,000 Layers. arXiv:2203.00555. https://arxiv.org/abs/2203.00555