Weight Pruning in Large Language Models: How SparseGPT, Wanda, and Semi-Structured 2:4 Sparsity Compress Neural Networks Without Retraining

Modern large language models require tens to hundreds of gigabytes of high-bandwidth memory to store billions of parameters. While post-training quantization compresses model footprints by reducing numerical precision from 16-bit floating point to 8-bit or 4-bit integers, weight pruning attacks model size along an orthogonal axis: setting redundant parameter values exactly to zero.

Historically, pruning deep neural networks required iterative cycles of magnitude thresholding and compute-heavy retraining. At the scale of 7B to 70B+ parameters, full retraining is computationally prohibitive. Over the past two years, algorithmic breakthroughs in post-training sparsification (most notably SparseGPT and Wanda) have made it possible to prune 50% of the weights in frontier models in minutes to hours on a single GPU without retraining, while maintaining near-dense perplexity.

The Failure of Magnitude Pruning in Large Transformers

In classical neural network compression, magnitude pruning ranks weights by their absolute value $|W_{ij}|$ and removes those closest to zero. The underlying assumption is that small weights exert minimal influence on layer activations.

In modern Transformer models, naive magnitude pruning degrades model performance rapidly once sparsity exceeds 10% to 20%. As documented by Tim Dettmers and colleagues in their study on LLM outliers, Transformers exhibit emergent activation outliers once model scale exceeds approximately 6.7 billion parameters. Specific hidden dimensions across sequence tokens develop activation magnitudes that are up to 100 times larger than average.

When an input activation $X_j$ is massive, even a near-zero weight $W_{ij}$ can produce a substantial contribution to the layer output $Y_i = \sum_j W_{ij} X_j$ . Setting that small weight to zero induces a severe perturbation $\Delta Y_i = - W_{ij} X_j$ , distorting subsequent layer representations and causing perplexity to explode.

SparseGPT: Second-Order Reconstruction at Scale

To prune large models without full backpropagation, Elias Frantar and Dan Alistarh developed SparseGPT (ICML 2023). SparseGPT frames pruning as a layer-wise quadratic reconstruction problem. Given a linear layer with weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ and a calibration dataset of input activations $X \in \mathbb{R}^{d_{\text{in}} \times N}$ , the goal is to find a sparse matrix $\hat{W}$ that minimizes squared output error:

$\min_{\hat{W}} \|W X - \hat{W} X\|_2^2 \quad \text{subject to} \quad \|\hat{W}\|_0 \le (1 - s) \cdot (d_{\text{out}} \cdot d_{\text{in}})$

where $s$ is the target sparsity fraction.

SparseGPT adapts the classical Optimal Brain Surgeon framework. The error incurred by removing a set of weights can be approximated via the second-order Taylor expansion using the empirical Hessian $H = 2 X X^T$ . To ensure numerical stability, a damping term $\lambda I$ is added:

$H = 2 X X^T + \lambda I$

Direct inversion of $H$ across millions of weights is computationally intractable. SparseGPT resolves this by processing weight columns sequentially. When column $j$ is pruned, unpruned weights in subsequent columns are updated using the inverse Hessian:

$\Delta W_{:, j+1:d_{\text{in}}} = - \frac{W_{:, j} - \hat{W}_{:, j}}{[H^{-1}]_{jj}} \cdot H^{-1}_{j, j+1:d_{\text{in}}}$

By computing the Cholesky decomposition of $H^{-1}$ once per layer and performing synchronized batch updates across output rows, SparseGPT prunes models up to 175 billion parameters (such as OPT-175B and BLOOM-176B) to 50% unstructured sparsity in approximately 4 hours on a single NVIDIA A100 GPU.

Wanda: Pruning by Weights and Activations

While SparseGPT achieves low perplexity degradation, computing and updating second-order inverse Hessians introduces runtime overhead and memory pressure. In late 2023, Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter introduced Wanda (NeurIPS 2023).

Wanda asks whether second-order weight compensation is necessary if the initial pruning metric directly accounts for activation outliers. Wanda defines the importance score $S_{ij}$ of each weight entry $W_{ij}$ as the product of its absolute magnitude and the L2-norm of its corresponding input feature vector:

$S_{ij} = |W_{ij}| \cdot \|X_j\|_2 = |W_{ij}| \cdot \sqrt{\sum_{k=1}^N X_{k, j}^2}$

The key mechanics of Wanda include:

Row-wise evaluation: Pruning thresholds are computed independently per output row. For each row $i$ , the lowest $s\%$ of weights according to score $S_{ij}$ are set to zero.
Zero weight updates: Unlike SparseGPT, Wanda performs no updates to the remaining non-zero weights.
Computational speed: Because it only requires a single forward pass over a small calibration batch (typically 128 sequences of 2,048 tokens from C4) to collect activation norms, Wanda prunes a 70B parameter model in under 10 minutes.

Empirical evaluations across LLaMA-1, LLaMA-2, and modern open-weight architectures show Wanda achieving perplexity scores competitive with, and in several configurations superior to, SparseGPT at 50% unstructured sparsity.

Sparsity Patterns: Unstructured vs. 2:4 Semi-Structured

In theoretical benchmarks, setting 50% of weights to zero cuts theoretical multiply-accumulate operations in half. In production hardware, however, the structure of sparsity determines whether memory and latency gains materialize.

Unstructured Sparsity

In unstructured sparsity, zeros are distributed arbitrarily across the weight matrix. While this flexibility preserves model accuracy at higher sparsity ratios (up to 50-60%), standard GPU Tensor Cores cannot execute sparse matrix multiplications efficiently without specialized indices (such as Compressed Sparse Row formats). Due to indexing overhead and memory access irregularity, unstructured sparsity typically requires 70% to 80% sparsity before delivering wall-clock latency improvements over dense matrix kernels.

N:M Semi-Structured Sparsity (2:4 Pattern)

To bridge the gap between theoretical sparsity and hardware execution, NVIDIA introduced hardware support for N:M semi-structured sparsity starting with the Ampere architecture (A100) and continuing through Hopper and Blackwell.

In a 2:4 sparsity pattern, exactly 2 out of every 4 contiguous elements along a matrix row must be zero. The hardware exploits this structure:

Storage compression: Only the 2 non-zero values (at 16-bit or 8-bit precision) and a 2-bit index metadata vector are stored in GPU High Bandwidth Memory, reducing memory footprint by nearly 50%.
Sparse Tensor Cores: Dedicated hardware units unpack the 2:4 metadata and feed non-zero elements into dense compute units, doubling theoretical arithmetic throughput (up to 2x peak TFLOPs).

Both SparseGPT and Wanda natively support 2:4 semi-structured pruning by enforcing the 2-out-of-4 constraint locally within each 4-element block during score sorting.

Structured Pruning: Shearing Transformer Architectures

While post-training weight pruning zeros individual matrix elements, structured pruning removes entire architectural components, such as attention heads, intermediate MLP hidden dimensions ( $d_{\text{ffn}}$ ), or full Transformer layers.

Structured pruning produces standard dense checkpoints that run on any standard inference engine (such as vLLM, SGLang, or TensorRT-LLM) without specialized sparse kernels.

LLM-Pruner (Ma et al., NeurIPS 2023) evaluates structural dependencies across coupled layers using first-order Taylor approximations, removing non-critical attention heads and MLP channels before applying parameter-efficient recovery tuning (LoRA).
Sheared LLaMA (Xia et al., ICLR 2024) introduced targeted structured pruning paired with dynamic batch loading. By pruning LLaMA-2 7B down to standardized architectures of 1.3B and 2.7B parameters, Sheared LLaMA matched the performance of models trained from scratch on over 1 trillion tokens while using only 3% of the pre-training compute budget (50 billion recovery tokens on SlimPajama).

Pruning and Quantization: Joint Compression

Pruning and quantization operate on complementary dimensions of efficiency. In production pipelines, teams increasingly combine both techniques:

2:4 Sparsity with 8-bit / 4-bit Weight Quantization: Applying SparseGPT or Wanda to enforce 2:4 sparsity, followed by GPTQ or AWQ quantization, yields models that are 75% smaller than the original FP16 baseline while benefiting from both memory bandwidth reduction and Sparse Tensor Core acceleration.
Precision vs Sparsity Pareto Frontiers: Research shows that for general reasoning and coding tasks, 4-bit dense models generally preserve more semantic fidelity than 8-bit 50% sparse models of equivalent memory footprint. However, when batch sizes grow and serving shifts from memory-bandwidth-bound decoding to compute-bound prefill, 2:4 sparse kernels provide superior throughput scaling.

Practical Implementation Rules

When applying post-training pruning to modern LLM checkpoints:

Calibration Data Diversity: Use at least 128 to 256 representative sequences (2,048 tokens each) covering diverse domains. Calibration sets dominated by narrow distributions cause activation norm miscalibration and elevated downstream perplexity.
Layer-by-Layer Sensitivity: Attention projection matrices ( $W_q, W_k, W_v, W_o$ ) are frequently more sensitive to aggressive pruning than feed-forward network up/down projections ( $W_{\text{gate}}, W_{\text{up}}, W_{\text{down}}$ ). Allocating non-uniform sparsity budgets (e.g., 40% in attention, 60% in MLP) preserves higher accuracy.
Post-Pruning Recovery: If compute budget allows, 1,000 to 5,000 steps of LoRA fine-tuning on domain data after Wanda or SparseGPT pruning eliminates the remaining perplexity gap at 50% sparsity.