Gradient Low-Rank Projection (GaLore): How Dynamic Subspaces Enable Full-Parameter LLM Pre-Training with Low-Rank Memory

Pre-training foundation large language models has historically required massive compute clusters, largely due to the memory footprint of optimizer states rather than the model weights themselves. While parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) freeze weights and introduce small adapter matrices, they fail when applied to pre-training from scratch because they restrict parameter updates to a fixed, static low-rank manifold.

Gradient Low-Rank Projection (GaLore), introduced by researchers from Caltech, UT Austin, and Meta, presents an alternative mathematical formulation. Rather than constraining the weight matrices to be low-rank, GaLore exploits the empirical fact that the weight gradients themselves reside in slowly changing low-rank subspaces during training. By projecting weight gradients into dynamic orthogonal subspaces via periodic singular value decomposition (SVD), GaLore maintains full-rank parameter expressivity while reducing optimizer state memory by up to 65.5% in standard precision and over 82% when paired with 8-bit quantization.

The Optimizer Memory Bottleneck in Pre-Training

To understand the memory challenge during neural network training, consider the memory allocation per parameter $\Phi$ in a standard 16-bit mixed-precision training pipeline using the AdamW optimizer:

1. Model Parameters (BF16/FP16): $2 \times \Phi$ bytes. For a 7-billion-parameter model, weights consume approximately 14 GB of VRAM.
2. Gradients (BF16/FP16): $2 \times \Phi$ bytes (14 GB for a 7B model).
3. Optimizer States (FP32 First and Second Moments): Standard AdamW tracks the running mean of gradients ($M_t$) and the running uncentered variance ($V_t$) in 32-bit floating-point format to ensure numerical stability. This requires $2 \times 4 \times \Phi = 8 \times \Phi$ bytes (56 GB for a 7B model).
4. Master Weights (FP32): Standard mixed-precision engines often maintain an FP32 copy of the weights for gradient accumulation, requiring an additional $4 \times \Phi$ bytes (28 GB).

Optimizer states alone account for more than half of the non-activation memory footprint during pre-training. While methods like ZeRO-1 shard optimizer states across distributed nodes, single-node or single-GPU training remains heavily bottlenecked.

+-------------------------------------------------------------------------+
|                7B Parameter Model Memory Footprint (BF16)               |
+-------------------------------------------------------------------------+
| Model Weights (BF16)       | 14 GB                                      |
| Gradients (BF16)           | 14 GB                                      |
| AdamW Moments M_t, V_t     | 56 GB  <-- Dominant memory bottleneck      |
| FP32 Master Weights        | 28 GB                                      |
+-------------------------------------------------------------------------+
| Total Parameter & State    | 112 GB (Excluding Activations)             |
+-------------------------------------------------------------------------+

Methods such as LoRA circumvent optimizer memory during fine-tuning by freezing the pre-trained weight matrix $W_0 \in \mathbb{R}^{m \times n}$ and training two low-rank matrices $A \in \mathbb{R}^{r \times n}$ and $B \in \mathbb{R}^{m \times r}$ such that $W = W_0 + BA$. However, during pre-training from scratch, $W_0$ is randomly initialized. Constraining $W$ to a static rank-$r$ parameterization restricts the model to a low-dimensional manifold, preventing it from learning the full-rank representations required for general language understanding.

The Theoretical Foundation of Low-Rank Gradients

The foundational insight behind GaLore is that while the weight matrix $W \in \mathbb{R}^{m \times n}$ must remain full-rank, the gradient matrix $G \in \mathbb{R}^{m \times n}$ has an intrinsic low-rank structure during backpropagation.

In a deep transformer layer $l$, the forward pass computes activations $f_l = \sigma(W_l f_{l-1})$. During backpropagation with a batch size $B$, the gradient of the loss with respect to the weight matrix $W_l$ is given by the outer product of the backpropagated error vectors $\delta_l \in \mathbb{R}^{B \times m}$ and the input activations $f_{l-1} \in \mathbb{R}^{B \times n}$:

$$G_l = \frac{\partial \mathcal{L}}{\partial W_l} = \delta_l^T f_{l-1}$$

Because the matrix product is formed by multiplying a transposed $m \times B$ matrix by a $B \times n$ matrix, the mathematical rank of the gradient matrix $G_l$ is upper-bounded by the batch size:

$$\text{rank}(G_l) \le \min(B, m, n)$$

Even with large cumulative batch sizes, empirical spectrum analysis reveals that the singular values of gradient matrices in transformers decay rapidly. A small set of dominant singular vectors captures the vast majority of the gradient energy, and these dominant directions evolve slowly across consecutive optimization iterations.

The GaLore Algorithm: SVD Subspace Projections

GaLore exploits gradient low-rank structure by projecting the full-rank gradient $G_t$ into an orthogonal low-rank subspace before passing it to the optimizer. The optimizer maintains its state tensors exclusively within this compact subspace.

1. Orthogonal Subspace Construction

For a weight matrix $W \in \mathbb{R}^{m \times n}$ with $m \ge n$, GaLore periodically performs Singular Value Decomposition on the gradient matrix $G_t \in \mathbb{R}^{m \times n}$:

$$G_t = U_t \Sigma_t V_t^T$$

The left projection matrix $P_t \in \mathbb{R}^{m \times r}$ is formed by extracting the first $r$ columns of $U_t$, corresponding to the $r$ largest singular values. By construction, $P_t$ has orthonormal columns:

$$P_t^T P_t = I_r$$

When $m < n$, a right projection matrix $Q_t \in \mathbb{R}^{n \times r}$ is constructed from the first $r$ columns of $V_t$ instead.

2. Gradient Projection and Low-Rank Optimizer Tracking

The full gradient $G_t$ is projected into the $r$-dimensional subspace:

$$R_t = P_t^T G_t \in \mathbb{R}^{r \times n}$$

Rather than tracking moments for the $m \times n$ matrix, the optimizer (such as AdamW) updates its first and second moment buffers directly on $R_t$:

$$M_t = \beta_1 M_{t-1} + (1 - \beta_1) R_t$$

$$V_t = \beta_2 V_{t-1} + (1 - \beta_2) R_t^2$$

$$\tilde{M}_t = \frac{M_t}{1 - \beta_1^t}, \quad \tilde{V}_t = \frac{V_t}{1 - \beta_2^t}$$

$$\Phi_t = \frac{\tilde{M}_t}{\sqrt{\tilde{V}_t} + \epsilon} + \lambda R_t$$

Here, $\Phi_t \in \mathbb{R}^{r \times n}$ represents the low-rank optimization step.

3. Full-Rank Weight Update

To apply the update to the actual model weights, $\Phi_t$ is projected back into the ambient $m \times n$ parameter space:

$$\Delta W_t = \alpha P_t \Phi_t \in \mathbb{R}^{m \times n}$$

$$W_{t+1} = W_t - \eta \Delta W_t$$

where $\eta$ is the learning rate and $\alpha$ is a constant scaling factor (typically $\alpha = \frac{1}{\sqrt{r}}$ or a fixed hyperparameter).

4. Periodic Subspace Rotation

A static subspace projection would constrain parameter updates to a fixed rank-$r$ slice. To achieve full-parameter exploration, GaLore updates the projection matrix $P_t$ every $T$ steps (typically $T \in [50, 200]$):

1. At iteration $t$ where $t \pmod T = 0$, compute a new SVD on $G_t$ to obtain $P_t$.
2. Reset or rescale the low-rank optimizer states $M_t$ and $V_t$ for the new subspace basis.
3. Continue standard gradient projection until the next interval $t + T$.

Because the subspace basis $P_t$ continuously rotates across the optimization trajectory, the sum of updates over $K$ intervals spans the full rank of $\mathbb{R}^{m \times n}$:

$$\sum_{t=0}^{K \cdot T} \Delta W_t = \sum_{k=0}^{K-1} \sum_{i=0}^{T-1} \alpha P_{k \cdot T} \Phi_{k \cdot T + i} \in \mathbb{R}^{m \times n}$$

This rotation allows GaLore to achieve the same expressive convergence as full-rank gradient descent while keeping the active optimizer memory footprint bounded to $O(rn)$ at every individual time step.

Memory Accounting: AdamW vs. LoRA vs. GaLore

The practical benefit of GaLore is demonstrated through direct memory accounting for a linear layer with dimensions $m \times n$ and low-rank dimension $r$:

| Method | Trainable Weights | Gradients | Optimizer States (FP32) | Total Layer Memory | | :--- | :--- | :--- | :--- | :--- | | Standard AdamW | $2mn$ bytes | $2mn$ bytes | $8mn$ bytes | $12mn$ bytes | | LoRA ($r$) | $2(mr + rn)$ bytes | $2(mr + rn)$ bytes | $8(mr + rn)$ bytes | $2mn + 12(mr + rn)$ bytes | | GaLore ($r$) | $2mn$ bytes | $2mn$ bytes | $8rn + 2mr$ bytes | $4mn + 8rn + 2mr$ bytes | | GaLore 8-Bit ($r$) | $2mn$ bytes | $2mn$ bytes | $2rn + 2mr$ bytes | $4mn + 2rn + 2mr$ bytes |

For a standard projection layer in a 7B model where $m = 4096, n = 4096$ and rank $r = 128$:

• Standard AdamW Optimizer States: $2 \times 4 \times 4096 \times 4096 = 134.22\text{ MB}$.
• GaLore Optimizer States ($r=128$): $2 \times 4 \times 128 \times 4096 = 4.19\text{ MB}$ (plus $1.05\text{ MB}$ for storing $P_t$), representing a 96.1% reduction in per-layer optimizer memory.

Single-GPU 7B Pre-Training

When combined with per-layer gradient checkpointing and per-layer weight updates (where gradients are computed, projected, applied, and freed immediately during the backward pass rather than holding the entire model's gradients simultaneously), memory usage drops substantially:

+-------------------------------------------------------------------------+
|             LLaMA-7B Pre-Training Memory on a Single GPU (VRAM)         |
+-------------------------------------------------------------------------+
| Baseline 16-bit AdamW     | ~58.0 GB (Requires 80GB A100 / H100)        |
| GaLore (FP32 Optimizer)   | ~31.2 GB                                    |
| GaLore 8-Bit AdamW        | ~21.8 GB                                    |
+-------------------------------------------------------------------------+
| Single NVIDIA RTX 4090    | 24.0 GB VRAM Capacity (Fits comfortably)    |
+-------------------------------------------------------------------------+

As demonstrated in the original paper benchmarks, an 8-bit GaLore configuration makes it possible to pre-train a 7B LLaMA model from scratch on a single 24 GB consumer GPU (such as an NVIDIA RTX 4090) without requiring distributed tensor parallelism or CPU parameter offloading.

Empirical Validation and Downstream Evaluation

To verify whether low-rank gradient projection harms pre-training convergence, the authors evaluated GaLore against full-rank AdamW baselines on the C4 dataset across 1B and 7B parameter architectures.

Pre-Training Perplexity on C4 (19.7B Tokens)

| Model Architecture | Optimizer | Rank ($r$) | Memory (Optimizer) | Validation Perplexity | | :--- | :--- | :--- | :--- | :--- | | LLaMA-1B | Standard AdamW | Full ($d=2048$) | 8.0 GB | 15.39 | | LLaMA-1B | LoRA | 128 | 0.9 GB | 19.42 | | LLaMA-1B | GaLore | 128 | 1.8 GB | 15.42 | | LLaMA-7B | Standard AdamW | Full ($d=4096$) | 56.0 GB | 14.61 | | LLaMA-7B | GaLore | 128 | 12.8 GB | 14.65 | | LLaMA-7B | GaLore 8-Bit | 128 | 6.5 GB | 14.67 |

The evaluation demonstrates that while LoRA suffers significant perplexity degradation during pre-training (19.42 vs 15.39 on LLaMA-1B), GaLore matches the perplexity of standard AdamW within 0.03 to 0.06 points while cutting optimizer memory by up to 88.4%.

Zero-Shot Downstream Benchmarks (LLaMA-7B Pre-Trained)

Downstream task evaluations on checkpoints pre-trained with GaLore confirm that representation quality transfers across evaluation suites:

| Benchmark | Standard AdamW Baseline | GaLore ($r=128$) | | :--- | :--- | :--- | | MMLU (5-shot) | 25.8% | 25.7% | | ARC-Challenge (0-shot) | 37.1% | 37.4% | | ARC-Easy (0-shot) | 66.8% | 66.5% | | HellaSwag (0-shot) | 62.4% | 62.1% | | PIQA (0-shot) | 75.3% | 75.1% | | WinoGrande (0-shot) | 63.8% | 64.0% |

Fine-Tuning Performance on GLUE

When applied to downstream fine-tuning on RoBERTa-Base across the GLUE benchmark suite, GaLore achieves an average score of 85.89, slightly outperforming standard full fine-tuning (85.75) and LoRA (85.61), demonstrating flexibility across both pre-training and adaptation phases.

Implementation and Systems Considerations

Integrating GaLore into existing PyTorch training pipelines requires minimal structural changes, as provided in the official open-source repository:

from galore_torch import GaLoreAdamW8bit

# Partition parameters into projected 2D layers and standard 1D vectors
galore_params = []
standard_params = []

for name, param in model.named_parameters():
    if param.ndim == 2 and "embed" not in name:
        galore_params.append(param)
    else:
        standard_params.append(param)

param_groups = [
    {"params": standard_params},
    {
        "params": galore_params,
        "rank": 128,
        "update_proj_gap": 200,
        "scale": 0.25,
        "proj_type": "std",
    },
]

optimizer = GaLoreAdamW8bit(param_groups, lr=0.01)

Several practical considerations govern deployment in production training jobs:

1. Subspace Update Gap ($T$): Setting $T=200$ steps balances computational overhead with subspace tracking. Recomputing the SVD every 200 steps accounts for less than 2% of overall wall-clock training time. Setting $T$ too low increases SVD overhead, while setting $T$ too high causes the projection basis to lag behind the shifting gradient trajectory.
2. Rank Selection ($r$): Empirical tests indicate that $r=128$ is sufficient for 1B to 7B models. For larger models (e.g. 13B to 70B), increasing $r$ to 256 or 512 preserves full convergence while still providing over 80% memory savings relative to full-rank optimizer states.
3. Compatibility with Distributed Training: GaLore operates independently on each rank during Data Parallel (DDP) and Fully Sharded Data Parallel (FSDP) training. When combined with FSDP, GaLore further reduces the communication and per-node optimizer memory envelope, enabling higher micro-batch sizes and better hardware utilization.

Summary

By identifying and exploiting the low-rank properties of weight gradients rather than constraining weight matrices directly, GaLore bridges the gap between memory-efficient adaptation and full-parameter pre-training. The ability to periodically re-anchor low-rank optimizer projections provides an effective mathematical mechanism for training modern foundation models under constrained GPU memory budgets.

Sources

• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (Zhao et al., ICML 2024 / arXiv:2403.03507)
• Official GaLore GitHub Repository (jiaweizzhao/GaLore)
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• PyTorch Fully Sharded Data Parallel (FSDP) Architecture Paper (Zhao et al., 2023)