Activation-Aware Weight Quantization (AWQ): Mathematical Foundations, Salient Weight Protection, and INT4 Tensor Core Execution

Large language models have transformed AI applications, but their deployment remains constrained by memory and compute barriers. A 70B parameter model in FP16 occupies ~140 GB of VRAM — exceeding even the 192 GB of NVIDIA's flagship B200 GPU, let alone edge devices. Quantization addresses this by reducing weight precision from 16-bit floats to 4-bit integers, shrinking model size by ~4×. However, naive rounding (Round-to-Nearest, RTN) catastrophically degrades model quality, especially at INT4 and below.

Activation-Aware Weight Quantization (AWQ), introduced by Lin et al. from MIT HAN Lab (arXiv:2306.00978, MLSys 2024 Best Paper Award), solves this through a simple but powerful insight: not all weights are equally important. By protecting only ~1% of "salient" weight channels — identified via activation magnitudes rather than weight magnitudes — AWQ achieves near-FP16 perplexity at INT4/INT3 precision without mixed-precision hardware overhead.

The Quantization Landscape

Quantization maps floating-point tensors to lower-bit integers. For LLMs, two settings dominate:

| Setting | Weights | Activations | Use Case | |---------|---------|-------------|----------| | W8A8 | INT8 | INT8 | Balanced compression, requires activation quantization | | W4A16 | INT4 | FP16 | Weight-only, maximum memory reduction, memory-bound decode acceleration |

AWQ targets weight-only grouped quantization (W4A16/INT3) — the only setting that simultaneously reduces memory footprint (enabling larger models on fixed hardware) and improves arithmetic intensity during autoregressive generation.

Why Weight-Only?

During autoregressive decoding, the model is memory-bound: arithmetic intensity (FLOPs/Byte) ≈ 1 on modern GPUs (peak ~165 TFLOPS / 1 TB/s bandwidth). Weight access dominates memory traffic by orders of magnitude over activation access. Quantizing weights 4× (FP16→INT4) raises arithmetic intensity to ~4 FLOPs/Byte, moving the workload toward compute-bound territory and unlocking 3-4× theoretical speedup.

Core Insight: Activation Magnitude Determines Weight Saliency

Standard importance metrics (weight L2 norm, magnitude) fail for LLMs. Lin et al. discovered that input activation magnitude identifies salient weight channels:

• Weight channels processing large-magnitude activations contribute disproportionately to layer output
• Quantization error on these channels is amplified by the activation magnitude
• Protecting ~1% of channels (by activation magnitude) recovers most of the FP16 perplexity

| Selection Method | 0.1% FP16 | 1% FP16 | 3% FP16 | |------------------|-----------|---------|---------| | Activation magnitude | ✓ Near-FP16 | ✓ Near-FP16 | ✓ Near-FP16 | | Weight magnitude | ✗ No improvement | ✗ Marginal | ✗ Marginal | | Random | ✗ No improvement | ✗ Marginal | ✗ Marginal |

Table: OPT-6.7B WikiText-2 perplexity under INT3-g128. Only activation-based selection works.

This is counterintuitive: weight-only quantization should look at activations, not weights, to determine which weights matter.

Mathematical Formulation: Per-Channel Scaling as Equivalent Transformation

Mixed-precision (keeping salient weights in FP16) is hardware-inefficient. AWQ instead applies an equivalent per-channel scaling transformation that reduces quantization error on salient channels while keeping all weights in INT4/INT3.

Quantization Error Analysis

For a weight group $\mathbf{w}$, standard symmetric quantization:

$$Q(\mathbf{w}) = \Delta \cdot \text{Round}\left(\frac{\mathbf{w}}{\Delta}\right), \quad \Delta = \frac{\max(|\mathbf{w}|)}{2^{N-1}}$$

The quantization error for element $w$ with input $x$:

$$\text{Err}(Q(w)x) = \Delta \cdot \text{RoundErr}\left(\frac{w}{\Delta}\right) \cdot x$$

where $\text{RoundErr}(\cdot) \sim 0.25$ (uniform distribution).

Now scale the salient weight by $s > 1$ and inversely scale the input:

$$\text{Err}(Q(w \cdot s)(x/s)) = \Delta' \cdot \text{RoundErr}\left(\frac{w \cdot s}{\Delta'}\right) \cdot x \cdot \frac{1}{s}$$

Key observation: Scaling $w$ rarely changes the group maximum, so $\Delta' \approx \Delta$. The error ratio becomes:

$$\frac{\text{Err}_\text{new}}{\text{Err}_\text{orig}} \approx \frac{1}{s} < 1$$

Thus scaling up salient weights by $s$ reduces their relative quantization error by $1/s$.

Optimal Scale Search

To balance salient vs. non-salient channels, AWQ searches for per-input-channel scaling factors $\mathbf{s}$ minimizing output reconstruction error:

$$\mathbf{s}^* = \arg\min_{\mathbf{s}} \mathcal{L}(\mathbf{s}) = \left\| Q(\mathbf{W} \cdot \text{diag}(\mathbf{s})) (\text{diag}(\mathbf{s})^{-1} \cdot \mathbf{X}) - \mathbf{W}\mathbf{X} \right\|$$

where $\mathbf{W}$ = original FP16 weights, $\mathbf{X}$ = cached activations from a small calibration set.

The quantization function is non-differentiable, so gradient-based optimization is unstable. AWQ constrains the search space using activation-awareness:

$$\mathbf{s} = \mathbf{s}_X^\alpha, \quad \alpha^* = \arg\min_\alpha \mathcal{L}(\mathbf{s}_X^\alpha)$$

where $\mathbf{s}_X$ = per-channel average activation magnitude, $\alpha \in [0, 1]$ balances protection strength. A fast grid search (20 steps) over $\alpha$ finds the optimum. Weight clipping further minimizes MSE.

AWQ vs. GPTQ vs. RTN: Key Differences

| Aspect | RTN | GPTQ | AWQ | |--------|-----|------|-----| | Method | Round-to-nearest | 2nd-order error compensation (Hessian) | Per-channel activation-aware scaling | | Calibration data | None | Large (128-192 sequences) | Tiny (16 sequences, 10× less) | | Backpropagation | No | No (but layer-wise reconstruction) | No | | Overfitting risk | Low | High (calibration-set dependent) | Minimal (activation stats only) | | Generalization | Poor at low bits | Domain-specific | Cross-domain, multi-modal | | Hardware format | Uniform INT4/INT3 | Uniform INT4/INT3 | Uniform INT4/INT3 | | Speed | Fastest | Slow (sequential layer processing) | Fast (parallelizable) |

Experimental Results

Base Models (LLaMA / Llama-2 / OPT)

| Model | FP16 | RTN (INT4) | GPTQ (INT4) | AWQ (INT4) | |-------|------|------------|-------------|------------| | LLaMA-7B | 5.68 | 5.96 | 6.22 | 5.78 | | LLaMA-13B | 5.09 | 5.25 | 5.23 | 5.19 | | LLaMA-30B | 4.10 | 4.23 | 4.24 | 4.21 | | LLaMA-65B | 3.53 | 3.67 | 3.66 | 3.62 | | Llama-2-70B | 3.32 | 3.46 | 3.42 | 3.41 |

WikiText-2 perplexity (↓). AWQ consistently beats GPTQ and RTN across scales.

Instruction-Tuned & Multi-Modal Models

AWQ is the first PTQ method to successfully quantize multi-modal LLMs without quality collapse:

| Model | Task | FP16 | RTN (INT4) | GPTQ (INT4) | AWQ (INT4) | |-------|------|------|------------|-------------|------------| | Vicuna-7B/13B | GPT-4 eval | — | ↓ | ↓ | ✓ Best | | CodeLlama-7B | MBPP pass@1 | 38.5 | 37.5 | 32.0 | 40.6 | | Llama-2-70B | GSM8K | 56.4 | 54.0 | 56.0 | 56.4 | | OpenFlamingo-9B | COCO CIDEr (32-shot) | 81.7 | 77.1 | 75.0 | 80.5 | | VILA-13B | 11 VLM benchmarks | — | — | — | Lossless |

Activation-aware scaling generalizes because it preserves the model's feature geometry, not just calibration-set statistics.

Data Efficiency & Robustness

• 10× smaller calibration set: AWQ reaches target perplexity with 16 sequences vs. GPTQ's 192
• Distribution shift resilience: Cross-domain calibration (PubMed→Enron) degrades AWQ by 0.5-0.6 PPL vs. GPTQ's 2.3-4.9

System Implementation: TinyChat — Realizing Theoretical Speedups

Quantization saves memory, but converting that to measured throughput requires systems work. TinyChat bridges this gap:

On-the-Fly Dequantization

Hardware lacks INT4×FP16 multiply instructions. TinyChat fuses dequantization into the GEMM kernel, avoiding DRAM writes of dequantized weights. Applied to both MM (prefill) and MV (decode) kernels.

SIMD-Aware Weight Packing

• ARM NEON (128-bit): Pack 32 INT4 weights per register using strided layout $w_0,w_{16},w_1,w_{17}...$ — 3 SIMD instructions vs. 96 scalar ops
• NVIDIA GPU: Pack 8 weights as $w_{\{0,2,4,6,1,3,5,7\}}$ for tensor core alignment

Kernel Fusion

• Fused LayerNorm (mul, div, sqrt in one kernel)
• Fused QKV projections + on-the-fly RoPE
• Pre-allocated KV cache with in-kernel updates
• Reduces kernel launch overhead (≈0.01 ms/kernel on RTX 4090)

Measured Speedups

| Platform | Model | FP16 (tokens/s) | AWQ INT4 (tokens/s) | Speedup | |----------|-------|-----------------|---------------------|---------| | RTX 4090 | Llama-2-7B | 52 | 162 | 3.1× | | RTX 4090 | Llama-2-13B | OOM | 99 | Enabled | | Jetson Orin | VILA-7B | 11.5 | 35.6 | 3.1× | | RTX 4070 (8GB) | Llama-2-13B | OOM | 33 | Enabled | | Raspberry Pi 4B | 7B model | — | 0.7 | Enabled |

Industry Adoption

AWQ has become the de facto standard for INT4 weight-only quantization across the inference stack:

• NVIDIA TensorRT-LLM: Native AWQ kernel support
• vLLM: vllm/model_executor/layers/quantization/awq.py
• Hugging Face Transformers: AutoAWQForCausalLM, AwqConfig
• AMD, Intel Neural Compressor, Google Vertex AI, AWS SageMaker, FastChat, LMDeploy, Hugging Face TGI
• 6M+ downloads of AWQ-quantized models on Hugging Face Hub
• Falcon-180B on single H200, Llama-2-70B on Jetson Orin 64GB

Limitations & When AWQ Isn't Enough

• INT2 and below: AWQ+GPTQ combination helps but quality degrades significantly
• Activation quantization (W4A4/W4A8): AWQ is weight-only; SmoothQuant / ZeroQuant address activation quantization
• Extreme compression: For <3-bit, QAT or distillation remains superior
• Non-transformer architectures: Designed for standard transformer linear layers

Conclusion

AWQ reframed LLM quantization from "uniform compression with error correction" to selective protection guided by activation statistics. Its contributions:

1. Saliency via activations: Identified that ~1% of weight channels, detectable through activation magnitudes, dominate quantization error
2. Hardware-friendly protection: Per-channel scaling achieves mixed-precision benefits in uniform INT4/INT3 format
3. Generalization without overfitting: Calibration-set agnostic activation statistics preserve cross-domain, multi-modal capabilities
4. Systems co-design: TinyChat demonstrates 3× real-world speedups on desktop, laptop, and mobile GPUs

AWQ established that intelligent quantization beats brute-force reconstruction — a principle now foundational to production LLM deployment at the edge and in datacenters.

Sources

1. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., & Han, S. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978v6. https://arxiv.org/abs/2306.00978
2. MIT HAN Lab. AWQ Project Page. https://hanlab.mit.edu/projects/awq
3. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323. https://arxiv.org/abs/2210.17323
4. Xiao, G., Lin, J., Seznec, M., Demouth, J., & Han, S. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438. https://arxiv.org/abs/2211.10438
5. Dettmers, T., & Zettlemoyer, L. (2022). The Case for 4-bit Precision: k-bit Inference Scaling Laws. arXiv:2212.09720. https://arxiv.org/abs/2212.09720
6. MIT HAN Lab. llm-awq GitHub Repository. https://github.com/mit-han-lab/llm-awq