Weight Quantization in Large Language Models: How GPTQ, AWQ, and SmoothQuant Compress Neural Networks

Serving large language models in production requires balancing GPU memory footprint, memory bandwidth limits, and computational throughput. A standard 70-billion parameter transformer stored in 16-bit floating-point (FP16 or BF16) requires roughly 140 gigabytes of high-bandwidth memory (HBM) merely to hold its parameters. This memory footprint exceeds the capacity of a single 80GB GPU before accounting for key-value (KV) caches, activation buffers, or batch concurrency.

Post-training quantization (PTQ) compresses pre-trained model weights from 16-bit representations down to 8-bit, 4-bit, or sub-4-bit numerical formats without retraining the model from scratch. Modern PTQ methods preserve generative accuracy through mathematical error compensation and activation-aware scaling.

Understanding the mechanics of modern quantization algorithms, including GPTQ, AWQ, and SmoothQuant, explains how modern inference frameworks achieve multi-fold compression and throughput gains across diverse hardware targets.

The Memory Bandwidth Bottleneck in LLM Generation

Inference in autoregressive transformer models consists of two distinct operational phases: prefill and decoding.

During the prefill phase, the model processes the entire prompt context concurrently. Matrix multiplications operate on matrices of shape $(B \times L, D)$, where $B$ is batch size, $L$ is prompt length, and $D$ is hidden dimension. This phase exhibits high arithmetic intensity (FLOPs per byte transferred) and is primarily compute-bound on modern GPU Tensor Cores.

During the decoding (token generation) phase, the model generates one token per sequence per forward pass. Matrix operations collapse to vector-matrix multiplications of shape $(B \times 1, D) \times (D, D_{\text{out}})$. At small batch sizes ($B=1$ to $4$), the arithmetic intensity drops below the GPU hardware compute-to-bandwidth balance point. The processor spends the majority of execution cycles waiting for weights to transfer from HBM into on-chip SRAM cache and registers.

+---------------------------------------------------------------+
|                LLM Inference Operational Regimes              |
+---------------------------------------------------------------+
|  Prefill Phase:                                               |
|  - Prompt processing: (Batch * SeqLen) x HiddenDim            |
|  - High Arithmetic Intensity (FLOPs/Byte > 150)               |
|  - Bottleneck: Tensor Core Compute Throughput                 |
+---------------------------------------------------------------+
|  Decode Phase:                                                |
|  - Token generation: (Batch * 1) x HiddenDim                  |
|  - Low Arithmetic Intensity (FLOPs/Byte < 10)                 |
|  - Bottleneck: High Bandwidth Memory (HBM) Transfer Speed     |
+---------------------------------------------------------------+

Because decoding throughput is constrained by memory bandwidth, reducing weight precision directly accelerates token generation. Loading a 4-bit integer weight consumes one-fourth the bandwidth of a 16-bit float. When weight-only quantization (W4A16) is used, weights are stored in 4-bit format in HBM, transferred to SRAM, and unpacked on-the-fly into FP16 registers before matrix arithmetic. This eliminates the bandwidth bottleneck without modifying core floating-point computation paths.

Quantization Mechanics and Numerical Representations

Quantization maps a continuous real-valued tensor $W \in \mathbb{R}$ to a discrete grid of integers $Q \in \mathbb{Z}$.

Uniform Affine Quantization

Uniform quantization applies a constant step size (scale factor $s$) and an optional zero-point offset ($z$):

$$q = \text{clamp}\left(\left\lfloor \frac{w}{s} \right\rceil + z, q_{\min}, q_{\max}\right)$$

The dequantized value $\hat{w}$ approximates the original float:

$$\hat{w} = s \cdot (q - z)$$

Where:

• Symmetric quantization sets $z = 0$, aligning the numerical zero of the integer grid with $0.0$. The scale factor is $s = \frac{\max(|w|)}{2^{b-1} - 1}$ for signed $b$-bit integers.
• Asymmetric quantization allows $z \neq 0$, mapping arbitrary dynamic ranges $[w_{\min}, w_{\max}]$ across the integer span $[0, 2^b - 1]$.

Quantization Granularity

Quantization error depends on the grouping granularity over which scale factors and zero points are shared:

1. Per-Tensor Quantization: A single scale factor $s$ is assigned to an entire parameter tensor (e.g., a $4096 \times 4096$ projection matrix). This approach exhibits minimal metadata overhead but suffers when weight magnitudes vary widely across channels.
2. Per-Channel Quantization: Each output row (or input column) receives an independent scale factor $s_i$. This accommodates variance across projection dimensions with negligible storage overhead.
3. Group-Wise Quantization: Rows are partitioned into small consecutive sub-vectors of group size $g$ (commonly $g = 64$ or $g = 128$), each receiving an independent scale and zero point. Group-wise quantization limits outlier propagation to local segments, preventing localized weight anomalies from degrading entire matrices at 3-bit and 4-bit precision.

The Activation Outlier Challenge

Naive Round-to-Nearest (RTN) quantization maps weights to the closest discrete bin independently. While RTN maintains acceptable perplexity at 8-bit precision, applying RTN at 4-bit causes significant degradation across generative tasks.

The primary driver of quantization degradation in transformers is the emergence of activation outliers. Research by Dettmers et al. (2022) demonstrated that once transformer models scale beyond 6.7 billion parameters, hidden state activations develop systematic, high-magnitude dimensions.

In approximately 0.1% of hidden channels, activation values reach magnitudes up to 100 times larger than the channel median. These outlier channels are not random noise:

• They concentrate in specific, consistent feature coordinates across transformer layers.
• They track critical contextual state and syntactic tokens.
• Quantizing these specific channels with coarse scaling destroys their dynamic range, causing severe model perplexity degradation.

Modern post-training quantization algorithms were developed to resolve this outlier dynamic through mathematical compensation and activation awareness.

GPTQ: Second-Order Error Compensation

Introduced by Frantar et al. (2022), GPTQ (Generative Pre-trained Transformer Quantization) frames weight quantization as an optimization problem over a small calibration dataset (typically 128 to 512 token sequences).

Instead of quantizing weights independently, GPTQ minimizes the squared output error between the original layer output and the quantized layer output:

$$\arg\min_{\hat{W}} \| W X - \hat{W} X \|_2^2$$

Where $W$ is the unquantized weight matrix and $X$ is the calibration activation matrix.

Optimal Brain Surgeon Formulation

GPTQ builds upon the Optimal Brain Surgeon (OBS) second-order Taylor expansion framework. When a specific weight element $w_q$ is rounded to discrete value $\hat{w}_q$, the rounding error is $\delta = \hat{w}_q - w_q$. To minimize the increase in layer reconstruction error, all remaining unquantized weights in the layer must adjust by a compensatory vector $\Delta W$:

$$\Delta W = - \frac{w_q - \hat{w}_q}{[H^{-1}]_{qq}} \cdot H^{-1}_{:, q}$$

Where $H = 2 X X^T$ is the empirical Hessian matrix representing input activation correlations, and $[H^{-1}]_{qq}$ is the $q$-th diagonal element of the inverse Hessian.

+---------------------------------------------------------------+
|                    GPTQ Column Quantization Loop              |
+---------------------------------------------------------------+
|  1. Compute Hessian H = 2 * X * X^T on calibration data       |
|  2. Invert Hessian: H_inv = (H + lambda * I)^(-1)             |
|  3. Decompose H_inv via Cholesky decomposition                |
|  4. For each block of columns:                                |
|     a. Quantize current column: w_q -> round(w_q / s) * s     |
|     b. Calculate quantization error: err = w_q - round(w_q)   |
|     c. Update remaining unquantized columns:                  |
|        W[:, unquantized] -= (err / H_inv[q, q]) * H_inv[:, q] |
|  5. Output calibrated low-bit weights                         |
+---------------------------------------------------------------+

Computational Innovations in GPTQ

Direct OBS execution requires recomputing or updating the inverse Hessian for every individual weight scalar, yielding prohibitive $O(d^3)$ complexity per layer. GPTQ introduces three algorithmic optimizations that enable full 175B model quantization in under four GPU hours:

1. Cholesky Pre-computation: The full inverse Hessian is computed once per layer. Using Cholesky decomposition, all weight updates for a column dimension are derived from pre-factored matrices.
2. Lazy Batch Updates: Updates to remaining columns are accumulated across blocks of 128 columns before being written back into memory using matrix-matrix multiplications (BLAS level 3 operations), saturating GPU compute resources.
3. Act-Order (Activation Ordering): Columns with larger activation variance are quantized first, allowing remaining parameters greater degrees of freedom to absorb critical reconstruction errors.

AWQ: Activation-Aware Weight Quantization

While GPTQ updates unquantized weights using inverse Hessian projections, Lin et al. (2023) demonstrated that second-order weight updates risk overfitting to the calibration dataset.

AWQ (Activation-aware Weight Quantization) takes an alternative structural approach based on a core empirical finding: not all weights are equally important.

Saliency and Channel Scaling

By analyzing activation distributions across calibration data, AWQ determines that protecting the top 0.1% to 1% of salient weight channels (those corresponding to high-magnitude activation channels) prevents nearly all quantization perplexity loss.

However, retaining 1% of weights in FP16 while quantizing 99% to INT4 (mixed-precision sparse storage) introduces hardware execution branch divergence and non-coalesced memory access.

AWQ resolves this by applying per-channel mathematical scaling factors before uniform quantization. Given a linear layer $Y = X W$, a diagonal scaling matrix $S = \text{diag}(s_1, s_2, \dots, s_d)$ is introduced:

$$Y = X W = (X S^{-1}) (S W) = X' W'$$

The transformed weight matrix $W' = S W$ and transformed activation matrix $X' = X S^{-1}$ produce identical mathematical outputs.

+---------------------------------------------------------------+
|               AWQ Channel Protection Mechanism                |
+---------------------------------------------------------------+
|                                                               |
|   Input Activation X (contains outlier channel i)             |
|   Scale factor s_i > 1.0 reduces activation magnitude:        |
|   X'_i = X_i / s_i                                            |
|                                                               |
|   Weight Matrix W (channel i scaled up by s_i):               |
|   W'_i = W_i * s_i                                            |
|                                                               |
|   Result: Relative quantization error on salient channel i    |
|   is reduced by factor of 1 / s_i during uniform INT4 rounding |
|                                                               |
|   Hardware Execution: Standard uniform INT4 GEMV kernel       |
|   with zero mixed-precision latency overhead                  |
+---------------------------------------------------------------+

Optimal Scale Search

Multiplying a salient weight column by $s_i > 1$ expands its dynamic range, which reduces relative quantization truncation error when mapping into discrete integer bins. Conversely, dividing activations by $s_i$ suppresses activation outlier peaks.

AWQ computes optimal scaling factors per channel by minimizing output reconstruction error via a constrained search space:

$$s = s_X^{\alpha} = \left(\frac{1}{N} \sum |X|\right)^\alpha$$

Where $\alpha \in [0, 1]$ is optimized via grid search on calibration sequences.

Because AWQ does not alter the fundamental numeric grid or introduce mixed-precision storage, AWQ-quantized models compile into uniform integer matrix multiplication kernels with zero runtime overhead.

SmoothQuant: Enabling W8A8 GEMM Acceleration

Both GPTQ and AWQ are primarily weight-only quantization schemes (W4A16 or W8A16), designed to eliminate memory bandwidth constraints during single-batch decoding.

When serving large concurrent batches or processing long prefill prompts, models become compute-bound. Accelerating compute-bound operations requires quantizing both weights and activations (W8A8), allowing matrix multiplications to execute directly on INT8 Tensor Cores.

Prior to SmoothQuant (Xiao et al., 2022), W8A8 suffered catastrophic accuracy degradation because activation outliers could not fit into uniform 8-bit integer dynamic ranges.

Migrating Quantization Difficulty

SmoothQuant observes an asymmetry in transformer layers:

• Activations are difficult to quantize due to sharp, localized channel outliers.
• Weights are easy to quantize because their values are uniformly distributed across channels.

SmoothQuant applies a per-channel smoothing scale $s \in \mathbb{R}^d$ across linear transformations, mathematically transferring quantization difficulty from activations to weights:

$$\hat{Y} = (X \cdot \text{diag}(s)^{-1}) \cdot (\text{diag}(s) \cdot W)$$

The migration scale factor balances maximum channel activation magnitude against maximum weight magnitude:

$$s_j = \frac{\max(|X_j|)^\alpha}{\max(|W_j|)^{1 - \alpha}}$$

Where $\alpha$ (migration strength) is typically set to $0.5$, evenly splitting the dynamic range difficulty between activations and weights.

+------------------------------------------------------------------------+
|                      SmoothQuant Transformation                        |
+------------------------------------------------------------------------+
|                                                                        |
|  Unsmoothed:                                                           |
|  Activations: [0.1, 0.2, 85.0, 0.3] -> Severe outlier on channel 3     |
|  Weights:     [0.05, 0.08, 0.04, 0.06] -> Uniform distribution         |
|                                                                        |
|  Apply Smoothing Scale s = [1.0, 1.0, 8.5, 1.0]:                       |
|  Smoothed Activations (X * s^-1): [0.1, 0.2, 10.0, 0.3]                |
|  Smoothed Weights     (s * W):    [0.05, 0.08, 0.34, 0.06]             |
|                                                                        |
|  Result: Both tensors quantize cleanly to INT8 with minimal truncation |
+------------------------------------------------------------------------+

Once smoothed, both activations and weights are quantized to INT8, enabling dense INT8 matrix multiplications (GEMM) across all linear projection layers.

Numerical Formats and Hardware Support

The choice of quantization method depends on the target deployment environment, batch concurrency, and GPU architecture.

+-----------------------------------------------------------------------------------------+
| Mode     | Method       | Weight Precision | Activation Precision | Primary Benefit     |
+----------+--------------+------------------+----------------------+---------------------+
| W4A16    | AWQ / GPTQ   | INT4 (group 128) | FP16 / BF16          | Memory reduction    |
| W8A8     | SmoothQuant  | INT8 (per-tensor)| INT8 (per-tensor)    | Compute speedup     |
| FP8      | Native FP8   | FP8 (E4M3)       | FP8 (E4M3/E5M2)      | Balanced speed & VRAM|
+-----------------------------------------------------------------------------------------+

FP8 Precision Scaling (Hopper, Ada Lovelace, Blackwell)

Recent hardware generations (such as NVIDIA H100/H200, L40S, and B200) introduce native hardware support for 8-bit floating point (FP8) arithmetic:

• E4M3 (1 sign bit, 4 exponent bits, 3 mantissa bits): Provides higher precision and bounded dynamic range, making it ideal for weights and activation forward passes.
• E5M2 (1 sign bit, 5 exponent bits, 2 mantissa bits): Matches standard FP16 dynamic range with reduced precision, used primarily for gradients and sensitive attention layers.

FP8 execution delivers up to a 2x throughput speedup on FP8 Tensor Cores while halving weight memory requirements compared to FP16, largely replacing complex integer calibration pipelines on Hopper and newer architectures.

Architectural Comparison and Trade-offs

| Criterion | Naive RTN | GPTQ | AWQ | SmoothQuant | Native FP8 | | :--- | :--- | :--- | :--- | :--- | :--- | | Precision Target | INT8 / INT4 | INT4 / INT3 | INT4 / INT3 | INT8 (W8A8) | FP8 (W8A8) | | Calibration Cost | None | Low (~1 hr) | Minimal (<10 min) | Minimal (<5 min) | Low (Online scaling) | | Weight Update Strategy | None | 2nd-order inverse Hessian | None (Channel scaling) | None (Channel scaling) | Scale factors only | | Generalization Risk | None | Slight (Dataset bias) | Very Low | Minimal | None | | Primary Acceleration Target | Memory capacity | Bandwidth / Decode | Bandwidth / Decode | Compute / Prefill & Batch | Compute & Bandwidth | | Hardware Requirement | Universal | Modern GPUs / CPUs | Modern GPUs / CPUs | INT8 Tensor Cores | Ada / Hopper / Blackwell |

Sources

• Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv preprint: https://arxiv.org/abs/2210.17323
• Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., & Han, S. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv preprint: https://arxiv.org/abs/2306.00978
• Xiao, G., Lin, J., Seznec, M., Demouth, J., & Han, S. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv preprint: https://arxiv.org/abs/2211.10438
• Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv preprint: https://arxiv.org/abs/2208.07339
• Micikevicius, P., Stosic, N., Venkatesh, B., Paris, T., et al. (2022). FP8 Formats for Deep Learning. arXiv preprint: https://arxiv.org/abs/2209.05433