Subscribe
AI
6 min read

What Is Quantization? How Language Models Shrink Without Breaking

Every AI model is, at bottom, a few billion numbers. Quantization is the practice of storing those numbers with fewer digits, the way you might write 3.14159 as 3.14 and rarely miss the difference. It is the single most consequential efficiency technique in the modern LLM economy: it is why a 70-billion-parameter model fits on one GPU, why open-weight models run on laptops, and why a fine-tune that once needed a server rack now needs a single card. It is also not free. This explainer covers wha

What Is Quantization? How Language Models Shrink Without Breaking

Every AI model is, at bottom, a few billion numbers. Quantization is the practice of storing those numbers with fewer digits, the way you might write 3.14159 as 3.14 and rarely miss the difference. It is the single most consequential efficiency technique in the modern LLM economy: it is why a 70-billion-parameter model fits on one GPU, why open-weight models run on laptops, and why a fine-tune that once needed a server rack now needs a single card.

It is also not free. This explainer covers what quantization does, where it breaks, and what the published research says about the cost.

The arithmetic that motivates everything

A model's weights are floating-point numbers, typically stored at 16 bits each (fp16 or bf16). Memory for the weights follows one rule: bytes is roughly the parameter count multiplied by bits per parameter, divided by eight.

  • A 7B model at fp16 needs about 14 GB. That already exceeds many consumer GPUs.
  • At 8-bit, the same model needs about 7 GB.
  • At 4-bit, about 3.5 GB, small enough for a mid-range laptop.
  • A 70B model halves to about 35 GB at 8-bit and lands near 35 to 40 GB at 4-bit, inside a single 80 GB datacenter GPU.

Hugging Face's Transformers documentation describes quantization in exactly these terms: lowering the precision of stored weights to cut memory while trying to preserve accuracy (docs).

The speed argument is subtler, and it is the reason quantization matters even when the model already fits in memory. During single-request decoding, a GPU spends most of its time fetching weight matrices from memory rather than doing arithmetic on them: the workload is memory-bandwidth-bound, not compute-bound (Machine Learning at Scale). Cutting bits per weight cuts the bytes fetched per token almost proportionally, so generation gets faster in roughly the same ratio.

How you actually shrink a number

Quantization maps a range of continuous values onto a small set of integers. The standard recipe is affine: pick a scale (and sometimes a zero-point offset), divide, round, store the integer plus the scale. Dequantizing multiplies back. Because the error depends on how well the scale matches the local distribution of values, modern methods store separate scales for small blocks of weights (per-group quantization) rather than one scale per layer.

The naive version, rounding to the nearest representable value with no feedback, is called "round-to-nearest." It works acceptably at 8 bits and degrades at 4 bits, which is why a decade of compression research got reinvented for LLMs.

Mapping continuous values onto a small set of integers

The three families of fixes

Compensating for the damage. GPTQ, from Frantar et al. 2022, quantizes weights column by column and uses second-order (Hessian) information from a small calibration set to redistribute each quantization error onto the weights that are still in full precision. It produces usable 4-bit models of 100B+ parameter scale on a single GPU in hours.

Protecting the important weights. AWQ, from Lin et al. 2023, observes that a small fraction of weight channels is salient, judged by the magnitude of the activations that flow through them. AWQ scales those channels up before quantization so their relative rounding error shrinks. No backpropagation, no Hessian, and it holds up better when the calibration data does not match deployment data.

Working around outliers. The failure mode that launched a subfield: at 175B scale, transformer activations develop a handful of enormous "emergent outlier" features that wreck uniform 8-bit quantization. LLM.int8() (Dettmers et al., NeurIPS 2022) isolates those outlier dimensions into a 16-bit matmul while keeping more than 99.9% of values in 8-bit, letting a 175B checkpoint load and run at int8 with no measured degradation. SmoothQuant (Xiao et al. 2022) attacks the same problem from the other side: it mathematically migrates outlier difficulty from activations into weights, enabling full 8-bit compute (W8A8) that uses integer tensor cores for batch throughput.

A different number format. NormalFloat 4 (NF4), introduced in QLoRA (Dettmers et al. 2023), is not an integer grid but a codebook chosen to be information-theoretically optimal for normally distributed weights. QLoRA paired NF4 with double quantization (compressing the quantization constants themselves) and paged optimizers, and showed a 65B model could be fine-tuned on a single 48 GB GPU while matching 16-bit fine-tuning quality. Its Guanaco models reached 99.3% of ChatGPT's level on the Vicuna benchmark after 24 hours on one GPU.

Separately, floating-point formats themselves moved down: FP8 (Micikevicius et al. 2022) established 8-bit float training and inference on Hopper-class hardware, and it is now the default serving precision for many large open models because it discards far less signal than integer formats at the same byte count.

On the consumer side, Georgi Gerganov's llama.cpp and its GGUF format, including the "K-quant" mixed schemes, turned quantization into a one-command experience and made running 4-bit open models on laptops and Apple Silicon a default rather than a project.

Weights are not the only memory

Long conversations live in the KV cache, which grows with context length and batch size and can rival the weights themselves. Quantizing the cache is a distinct problem because keys and values behave differently: keys have consistent per-channel structure while values vary per token. KIVI (Liu et al. 2024) used that asymmetry, per-channel for keys and per-token for values, to push the cache toward 2 bits with small accuracy loss. Practically, 8-bit or FP8 KV caches are now routine ways to serve longer contexts on the same hardware.

What the quality cost actually looks like

Precision traded for size on a scale

The honest summary of the published measurements:

  • 8-bit is nearly free, and size buys robustness. A comprehensive evaluation of instruction-tuned models from 1B to 405B parameters across four quantization methods and 13 datasets found that 70B-scale models hold stable performance even at 4 bits while small models suffer severe accuracy drops, and that FP8 is the most robust format across tasks. LLM.int8() reported no degradation at 175B.
  • 4-bit is close on knowledge, further from perfect on reasoning and code. An EMNLP 2025 study of long-context tasks found 8-bit methods lose under 3% relative versus BF16, while GPTQ-int4 and bitsandbytes NF4 lose at least 3% on all tested models and over 10% on several, with the damage concentrated at 64K-128K context lengths. A single-GPU benchmark of Qwen3-32B measured FP8 losing 0.6 points on MMLU-Pro versus BF16, while INT4 bought roughly 2.7x throughput but dropped about 8 points on HumanEval code generation.
  • Smaller models suffer more. Quantization error compounds where there is less redundancy, which is why 4-bit hits a 3B model harder than a 70B one (the 1B-to-405B evaluation documents the size-dependence directly).

The practitioner consensus that follows from this: quantize weights aggressively, keep precision where it matters (the first and last layers, attention projections, whatever your task is sensitive to), and measure on your workload, not on perplexity alone.

The one-line version

Quantization trades numeric precision for memory, bandwidth, and cost. At 8 bits the trade is close to invisible. At 4 bits it is cheap for most tasks and expensive for the ones that concentrate on code, math, or very long context. It is the reason the open-model ecosystem runs on hardware people can actually own.

Sources

AILLMeducational