Vector Quantization for LLM Weights in Production: Comparing QuIP#, AQLM, and VPTQ Architecture, Dequantization Kernels, and 2-Bit Serving Economics

Scalar post-training quantization methods such as GPTQ and AWQ have become the standard for compressing large language models to 4-bit integer formats (INT4). At 4 bits per parameter, scalar techniques preserve over 98% of baseline 16-bit floating-point (FP16/BF16) model accuracy across common benchmarks. However, pushing scalar quantization below 4 bits (into 2-bit or 3-bit regimes) causes severe perplexity degradation and catastrophic task failures.

To break past this barrier, modern weight-compression research has shifted toward Vector Quantization (VQ). Rather than rounding individual weight scalars independently onto a 1D grid, vector quantization algorithms map multidimensional blocks of weights simultaneously to discrete vector codebooks. Frameworks like QuIP#, AQLM, and VPTQ allow 70-billion-parameter models to operate within 18 to 20 GB of VRAM at approximately 2 bits per parameter, enabling single-accelerator deployments that were previously impossible.

This guide analyzes the theoretical foundations of vector quantization, compares the core architectures of QuIP#, AQLM, and VPTQ, examines dequantization kernel overheads during inference, and evaluates the economics of 2-bit serving in production environments.

Why Scalar Quantization Breaks Below 4 Bits

Scalar quantization techniques map a continuous weight scalar to a discrete integer index using a scale factor and optional zero-point offset. While effective at 8-bit and 4-bit precision, scalar quantization encounters three fundamental limitations at 2-bit and 3-bit precision:

1. Information-Theoretic Quantization Noise: In 1D scalar space, partitioning continuous distributions with only 4 discrete levels (2 bits) creates wide quantization intervals. According to rate-distortion theory, optimal 1D sphere packing leaves substantial unoccupied volume between scalar boundaries.
2. Channel Outlier Interference: In transformer weight matrices, a small fraction (often under 0.1%) of weight channels and activation dimensions exhibit extreme magnitudes. In scalar quantization, the dynamic range of the entire channel is scaled by these outliers, collapsing the resolution of the remaining 99.9% of weights.
3. Second-Order Hessian Sensitivity: Quantization error in one weight induces output degradation proportional to the layer input covariance (the Hessian matrix). Scalar methods like GPTQ sequentially adjust remaining unquantized weights via Cholesky updates to compensate for error, but at 2-bit precision, the accumulated residual error overwhelms the network compensatory capacity.

Theoretical Foundations: Scalar vs. Vector Quantization

Vector quantization resolves these geometric limitations by grouping $d$ consecutive weights into a single vector $\mathbf{w} \in \mathbb{R}^d$ and quantizing the entire tuple to the closest centroid $\mathbf{c}_i$ in a precomputed codebook $\mathcal{C} = \{\mathbf{c}_1, \dots, \mathbf{c}_K\}$ of size $K = 2^B$.

The effective bitrate per parameter is:

$$b = \frac{B}{d} = \frac{\log_2 K}{d}$$

For example, selecting a vector dimension $d=8$ and a codebook size $K = 2^{16} = 65,536$ yields an effective bitrate of $16 / 8 = 2.0$ bits per parameter.

Key Advantages of High-Dimensional Quantization

• Lattice Packing Efficiency: In higher dimensions ($d \ge 8$), vector quantization partitions space into Voronoi polyhedra rather than hypercubes. Optimal lattices (such as the 8-dimensional Gosset lattice $E_8$ or the 24-dimensional Leech lattice $\Lambda_{24}$) fill Euclidean space with significantly lower mean squared error per dimension than independent 1D grids.
• Cross-Weight Correlation Capture: Weights within neural network layers exhibit subtle directional correlations. Vector quantization naturally models joint weight distributions without requiring independent coordinate axes.
• Outlier Incoherence: By projecting weight matrices into orthogonal subspaces prior to quantization, extreme outliers are dispersed across all dimensions, neutralizing the dynamic range collapse typical of scalar quantization.

Comparative Architectural Analysis

Three primary vector quantization frameworks have emerged for production LLM compression: QuIP#, AQLM, and VPTQ.

Core Architecture Comparison

• QuIP# (Cornell / Yandex)
• Quantization Mechanism: Incoherence processing + $E_8$ Lattice quantization
• Outlier Handling: Randomized Hadamard transform (RHT) on weight and activation matrices
• Codebook Structure: Fixed 8D Gosset lattice ($E_8$) with per-channel scaling
• Fine-Tuning Required: Post-quantization vector fine-tuning (PV-tuning)
• Effective Bitrate Range: 2.0 to 4.0 bits/parameter
• Compression Time (70B Model): ~4 to 8 GPU hours (Nvidia A100)
• WikiText-2 Perplexity (Llama-2 70B @ 2-bit): ~5.8 to 6.1
• AQLM (Yandex / IST Austria)
• Quantization Mechanism: Additive Vector Quantization (multi-codebook residual sums)
• Outlier Handling: Joint optimization over learned centroids and scaling factors
• Codebook Structure: 1 to 2 learned codebooks of dimension $d=8$ ($2^{16}$ centroids each)
• Fine-Tuning Required: End-to-end differentiable codebook updates across layers
• Effective Bitrate Range: 2.0 to 3.0 bits/parameter
• Compression Time (70B Model): ~12 to 24 GPU hours (Nvidia A100)
• WikiText-2 Perplexity (Llama-2 70B @ 2-bit): ~5.9 to 6.3
• VPTQ (Microsoft Research)
• Quantization Mechanism: Second-order Residual Vector Quantization (RVQ)
• Outlier Handling: Hessian-guided residual centroid correction
• Codebook Structure: Multi-stage residual codebooks with adaptive vector dimensions
• Fine-Tuning Required: None (pure analytical post-training optimization)
• Effective Bitrate Range: 1.8 to 4.0 bits/parameter
• Compression Time (70B Model): ~2 to 4 GPU hours (Nvidia A100)
• WikiText-2 Perplexity (Llama-2 70B @ 2-bit): ~5.6 to 6.0

1. QuIP#: Incoherence Processing and Lattice Codebooks

QuIP# builds on the theoretical principle of incoherence processing. Standard LLM weight and activation matrices are highly coherent, meaning a tiny subset of entries carries a disproportionately large fraction of the total matrix energy.

QuIP# applies randomized orthogonal Hadamard transformations to both the inputs and outputs of each linear layer:

$$\tilde{W} = U W V^T, \quad \tilde{X} = V X$$

where $U$ and $V$ are Kronecker products of randomized Walsh-Hadamard matrices. Because orthogonal transformations preserve Euclidean inner products ($\tilde{X}^T \tilde{W}^T = X^T W^T$), the underlying layer computation remains unchanged. Crucially, the Hadamard transform spreads outlier energy evenly across all coordinates, guaranteeing that no individual coordinate dominates.

After transformation, QuIP# quantizes 8-dimensional weight vectors using the $E_8$ Gosset lattice, the densest known sphere packing in 8 dimensions. To recover residual degradation, QuIP# introduces PV-tuning (Post-quantization Vector fine-tuning), which fine-tunes continuous scale factors and residual vectors against calibration sequences without altering the discrete lattice assignments.

2. AQLM: Additive Multi-Codebook Vector Quantization

AQLM (Additive Quantization for Large Models) approaches vector quantization through multi-codebook decomposition. Instead of mapping a weight vector $\mathbf{w} \in \mathbb{R}^d$ to a single codebook centroid, AQLM approximates $\mathbf{w}$ as the sum of $M$ sub-vectors selected from $M$ distinct codebooks $\mathcal{C}_1, \dots, \mathcal{C}_M$:

$$\hat{\mathbf{w}} = \sum_{m=1}^M \mathbf{c}_{m, i_m}, \quad \mathbf{c}_{m, i_m} \in \mathcal{C}_m$$

For a 2-bit configuration, AQLM typically employs $M=1$ or $M=2$ codebooks of dimension $d=8$ with $K = 2^{16}$ centroids, stored in 16-bit float precision.

The primary strengths of AQLM include:

• Beam Search Discrete Assignment: During quantization, AQLM runs joint beam search over all codebook permutations to minimize layer-wise reconstruction loss with respect to the input Hessian.
• Differentiable End-to-End Fine-Tuning: AQLM updates both the continuous codebook centroids and the layer normalization parameters across the entire network using straight-through estimators (STE), achieving competitive perplexity at 2.01 bits per parameter.

3. VPTQ: Second-Order Residual Vector Quantization

VPTQ (Vector Post-Training Quantization), developed by Microsoft Research, eliminates the expensive end-to-end backpropagation required by AQLM and QuIP# while extending vector quantization down to sub-2-bit regimes (1.8 to 2.2 bits).

VPTQ uses second-order optimization and Residual Vector Quantization (RVQ) across matrix columns:

1. Centroid Initialization: Initial codebook centroids are selected via Hessian-weighted k-means clustering.
2. Sequential Residual Quantization: Each column is quantized into a primary codebook centroid, and the residual error is quantized into a secondary residual codebook.
3. Hessian Error Tracking: Similar to Optimal Brain Surgeon (OBS) updates, VPTQ continuously updates the unquantized weights in remaining channels using inverse Hessian slices ($H^{-1}$), canceling cumulative quantization errors across long vector sequences.

Because VPTQ relies on analytical second-order updates rather than gradient-based fine-tuning, it compresses a 70B model in 2 to 4 GPU hours, compared to 12 to 24 hours for AQLM.

Dequantization Kernels and Inference Dynamics

While vector quantization delivers superior mathematical compression, serving performance depends directly on hardware dequantization throughput.

                  ┌────────────────────────────────────────┐
                  │      VRAM: 2-Bit Vector Indices        │
                  │   (e.g., 16-bit uint per 8 weights)    │
                  └──────────────────┬─────────────────────┘
                                     │ Memory Bandwidth (~5x less than FP16)
                                     ▼
                  ┌────────────────────────────────────────┐
                  │    GPU Streaming Multiprocessor (SM)   │
                  │                                        │
                  │  ┌──────────────────────────────────┐  │
                  │  │ Shared Memory / L1 Cache:        │  │
                  │  │ Small Codebooks (LUT)            │  │
                  │  └────────────────┬─────────────────┘  │
                  │                   │ Fast Lookup         │
                  │                   ▼                     │
                  │  ┌──────────────────────────────────┐  │
                  │  │ Dequantized FP16/BF16 Vectors    │  │
                  │  └────────────────┬─────────────────┘  │
                  │                   │                     │
                  │  ┌────────────────▼─────────────────┐  │
                  │  │ GEMV / Tensor Core Dot Products  │  │
                  │  │ with Input Activations (X)       │  │
                  │  └────────────────┬─────────────────┘  │
                  └───────────────────┼────────────────────┘
                                      │
                                      ▼
                  ┌────────────────────────────────────────┐
                  │         Output Logits / Tokens         │
                  └────────────────────────────────────────┘

1. Prefill vs. Decode Bottlenecks

LLM inference operates in two distinct compute regimes:

• Prefill Phase (Prompt Processing): Compute-bound matrix multiplication (GEMM). Because batch size and prompt length are large, arithmetic intensity is high. Vector dequantization introduces table lookup overhead that can reduce prefill throughput relative to hardware-accelerated FP16 or INT8 GEMMs.
• Decode Phase (Token Generation): Memory-bandwidth-bound matrix-vector multiplication (GEMV). For batch sizes $B=1$ to $B=8$, GPU execution time is dominated by reading weights from High Bandwidth Memory (HBM). By shrinking weight memory by 75% compared to FP16, vector quantization provides a direct 2x to 3.5x reduction in decode memory traffic.

2. Lookup Table (LUT) Overhead vs. Bit-Shifting

In scalar integer quantization (INT4/INT8), dequantization requires simple arithmetic bit-shifts, masking, and multiply-adds (fma), which execute directly on ALU registers.

In contrast, vector quantization requires index-based codebook lookups:

• Small Codebooks ($K \le 256$): Codebooks fit inside fast GPU shared memory or L1 cache. Lookups use warp shuffle primitives or local memory gather operations.
• Large Codebooks ($K \ge 65,536$): In AQLM and QuIP# 2-bit modes, large codebooks require table lookup instructions (such as ARM TBL or CUDA shared-memory indexing). If codebook lookups spill into L2 cache, memory latency can degrade the bandwidth savings gained from 2-bit weights.
• Fused Dequantization-GEMV Kernels: Production VQ runtimes (such as VPTQ CUDA/Triton kernels and AQLM fast dequantizers) fuse codebook lookup and scalar accumulation into single kernel launches, ensuring weights remain in registers without intermediate FP16 global memory writes.

Production Serving Economics

The economic case for 2-bit vector quantization centers on hardware tier compression: enabling larger models to run on lower-cost compute instances.

Memory Footprint Comparison for Llama-3 70B

• FP16 / BF16 (16.0 bits/param)
• Weight VRAM: ~140 GB
• Minimum Hardware: 2x Nvidia A100 (80GB) or 4x Nvidia A10G (24GB)
• Typical Cloud Cost: ~$6.50 to $8.00 / hour
• FP8 / INT8 (8.0 bits/param)
• Weight VRAM: ~70 GB
• Minimum Hardware: 1x Nvidia A100 (80GB) or 2x Nvidia L40S (48GB)
• Typical Cloud Cost: ~$3.50 to $4.50 / hour
• AWQ / GPTQ (4.0 bits/param)
• Weight VRAM: ~38 GB
• Minimum Hardware: 1x Nvidia A100 (40GB) or 2x Nvidia RTX 4090 (24GB)
• Typical Cloud Cost: ~$2.00 to $3.00 / hour
• QuIP# / AQLM (2.0 bits/param)
• Weight VRAM: ~19 GB
• Minimum Hardware: 1x Nvidia RTX 4090 (24GB) or 1x Nvidia A10G (24GB)
• Typical Cloud Cost: ~$0.75 to $1.20 / hour
• VPTQ (1.8 bits/param)
• Weight VRAM: ~17 GB
• Minimum Hardware: 1x Nvidia RTX 3090 / 4090 (24GB)
• Typical Cloud Cost: ~$0.50 to $1.00 / hour

Throughput and Concurrency Trade-Offs

Deploying 2-bit vector quantized models in production involves clear architectural trade-offs:

1. Low-Concurrency and On-Device Deployment ($B = 1$ to $4$): Vector quantization delivers maximum cost efficiency. A 70B parameter model can be served locally on a single consumer GPU or workstation (24GB VRAM) at generation speeds of 25 to 35 tokens per second.
2. High-Throughput Batch Serving ($B \ge 32$): As concurrency scales, inference transitions from memory-bound to compute-bound. Tensor core-optimized formats (such as FP8 with native Hopper/Blackwell matrix engines or INT4 with Marlin kernels) outperform VQ table lookups in total token generation throughput per second.

Engineering Decision Framework

                        Low-Bit Model Serving Decision
                                      │
               Is your primary bottleneck VRAM capacity
                    or low-concurrency latency?
                                      │
                     ┌────────────────┴────────────────┐
                    Yes                               No
                     │                                 │
         Can your workload tolerate          Deploy Native FP8 /
           2-3% perplexity loss vs FP16?      Marlin INT4 (Max Batch Throughput)
                     │
          ┌──────────┴──────────┐
         Yes                    No
          │                      │
   Do you require fast     Deploy 4-Bit AWQ / GPTQ
   offline compression?
          │
     ┌────┴────┐
    Yes        No
     │          │
 Deploy     Deploy QuIP# / AQLM
  VPTQ       (Max 2-Bit Perplexity Recovery via Fine-Tuning)

• Deploy VPTQ when compressing very large foundation models (70B, 405B) within tight compute budgets (hours instead of days) and deploying on single-GPU instances where memory capacity is the binding constraint.
• Deploy QuIP# or AQLM when serving long-term production endpoints where offline fine-tuning compute (PV-tuning) can be amortized over millions of requests to extract optimal 2-bit accuracy.
• Deploy Native FP8 or INT4 Marlin for multi-tenant high-throughput API clusters operating at continuous batch sizes above 32 concurrent requests.

Sources

• Tseng, A., et al. (2024). QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. arXiv: 2402.04396
• Egiazarian, V., et al. (2024). Extreme Compression of Large Language Models via Additive Quantization. arXiv: 2401.06118
• Wang, Y., et al. (2024). VPTQ: Extreme Low-Bit Vector Post-Training Quantization for Large Language Models. arXiv: 2409.17066
• Vanhoucke, V., et al. (2024). GPTVQ: The Blessing of Dimensionality for LLM Quantization. arXiv: 2402.15319
• Chee, J., et al. (2023). QuIP: 2-Bit Quantization of Large Language Models with Guarantees. arXiv: 2307.13304
• Frantar, E., et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv: 2210.17323
• Lin, J., et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv: 2306.00978