KV Cache Quantization in Production LLM Serving: Comparing FP8, INT8, INT4, KIVI, and KVQuant

In high-throughput large language model serving, the primary hardware bottleneck during autoregressive token generation is memory bandwidth and GPU High Bandwidth Memory (HBM) capacity rather than raw compute FLOPS. While model weights remain static and are loaded once per forward pass, the Key-Value (KV) cache grows dynamically with sequence length and batch size.

For a standard model in FP16 or BF16 precision, storing the KV cache requires 4 bytes per token across all layers and heads:

Memory per Token = 2 × 2 × n_layers × n_kv_heads × d_head bytes

For a model like Llama 3 70B (80 layers, 8 KV heads with Grouped-Query Attention, head dimension 128), a single sequence at a 128,000-token context length consumes approximately 10.48 GB of GPU memory. Serving a concurrent batch of 16 such sequences requires over 167 GB of HBM exclusively for the KV cache, exceeding the total memory of two 80 GB NVIDIA H100 GPUs before accounting for model weights.

To serve long contexts at high batch sizes, inference runtimes such as vLLM, SGLang, and TensorRT-LLM increasingly rely on KV cache quantization. Compressing KV cache activations from 16-bit precision to 8-bit, 4-bit, or 2-bit representations reduces memory footprints by 2x to 8x, directly scaling serving throughput by amortizing memory bandwidth traffic. However, quantizing dynamic KV activations introduces distinct numerical stability challenges that differ fundamentally from static weight quantization.

The Numerical Dilemma: Key Outliers vs. Value Dynamic Range

Directly applying standard post-training quantization (PTQ) techniques—such as uniform per-tensor or per-token INT4 quantization—to the KV cache leads to severe accuracy degradation on long-context retrieval and reasoning benchmarks. Research into activation distributions reveals that Key and Value tensors exhibit sharply contrasting numerical characteristics.

Key activations develop systematic, channel-specific outliers across layers. Certain feature dimensions maintain magnitudes up to 100 times larger than the average activation across the entire sequence. Because Rotary Position Embeddings (RoPE) rotate these activations along coordinate pairs, the outlier channels rotate across positions, preventing simple static channel scaling. When quantized uniformly per-token, the massive dynamic range of these persistent outlier channels crushes the quantization resolution of the remaining 99% of normal channels, destroying attention score precision.

Conversely, Value activations show high variance across token positions rather than fixed channels. Value vectors do not undergo RoPE rotation, but their magnitudes fluctuate dynamically across generation steps. Applying per-channel quantization to Value tensors introduces clipping errors across differing tokens.

Consequently, effective KV cache compression architectures must handle Key and Value tensors asymmetrically or alter the representation space prior to low-bit discretization.

Precision Levels and Architectural Approaches

1. FP8 KV Cache (E4M3 vs. E5M2)

FP8 is the standard low-precision KV cache format in production environments equipped with NVIDIA Hopper (H100), Blackwell (B200), or AMD CDNA3 (MI300X) accelerators.

• Numerical Format: Runtimes predominantly deploy the OCP FP8 E4M3 format (1 sign bit, 4 exponent bits, 3 mantissa bits) for cached activations due to its higher precision compared to E5M2.
• Scaling Strategies: Systems implement either per-tensor uncalibrated scaling (setting scale = 1.0) or offline dataset-calibrated scaling per attention head. Calibrated scales prevent underflow in deep layers where activation norms shift.
• Execution Pipeline: Modern attention kernels, including FlashAttention-3 and FlashInfer, execute native FP8 Matrix-Matrix multiplications (GEMM) directly on Tensor Cores. Because tensors remain in FP8 throughout storage and computation, there is zero dequantization latency overhead.
• Operational Profile: FP8 halves KV cache memory (1 byte per element), preserving 97% to 99% of full-precision retrieval accuracy up to 128k context lengths without requiring specialized kernel compilation or outlier filtering.

2. INT8 and INT4 Uniform Group-Wise Quantization

On hardware lacking native FP8 Tensor Core support (such as NVIDIA Ampere A100 or Ada Lovelace architectures), integer quantization provides a portable alternative.

• INT8 Per-Token/Per-Channel: INT8 symmetric quantization maps values to [-127, 127] using a single scale factor per group. It reduces memory by 50% with near-zero downstream task degradation.
• INT4 Group-Wise: Compresses elements to 4 bits with group sizes typically set to 32, 64, or 128 elements along the channel dimension. Grouping isolates channel outliers to specific sub-vectors.
• Kernel Overhead: Unlike FP8 on modern hardware, INT4 attention requires dequantizing cached blocks back to FP16 in GPU SRAM before computing query-key dot products, introducing memory bandwidth and compute trade-offs in register-heavy attention kernels.

3. KIVI: Tuning-Free Asymmetric 2-Bit/4-Bit Quantization

Introduced at ICML 2024, KIVI introduces an asymmetric quantization framework tailored specifically to the divergent distributions of Keys and Values.

• Key Cache Strategy: KIVI quantizes the Key cache per-channel across all sequence tokens, matching the dimensional persistence of key outliers.
• Value Cache Strategy: KIVI quantizes the Value cache per-token along the hidden dimension, aligning with token-level variance.
• Residual FP16 Buffer: To avoid quantizing recent, highly dynamic tokens, KIVI maintains a small streaming residual window (e.g., the most recent 32 to 128 tokens) in full FP16 precision. When tokens exit the window, they are quantized and packed into memory blocks.
• Compression: KIVI achieves tuning-free 2-bit and 4-bit KV cache storage. While 4-bit KIVI matches FP16 performance across standard evaluation suites, 2-bit quantization exhibits modest degradation on multi-step reasoning benchmarks (such as MATH and GPQA), but cuts peak memory by 2.6x and boosts throughput by up to 3.47x.

4. KVQuant: Outlier-Aware Non-Uniform Quantization

Published at NeurIPS 2024, KVQuant combines non-uniform quantization with sparse outlier preservation to scale KV cache compression down to 3 bits and 2 bits.

• Pre-RoPE Quantization: KVQuant quantizes Key vectors prior to applying RoPE. Because RoPE mixes channel coordinates dynamically, quantizing pre-RoPE activations isolates channel outliers to static dimensions.
• Non-Uniform Quantization (NUQ): Instead of uniform integer bins, KVQuant uses sensitivity-weighted, non-uniform quantization grids derived via k-means clustering over calibration activations.
• Dense-and-Sparse Matrix Decomposition: The top 1% of extreme outlier activations are retained in a sparse FP16 matrix, while the remaining 99% of dense values are packed into 3-bit or 4-bit representations.
• Target Domain: This approach enables context lengths of up to several million tokens on single-node GPU clusters, though it requires custom CUDA decoding kernels to reconstruct sparse-dense dot products.

5. QuaRot and RotateKV: Orthogonal Hadamard Dispersal

Rather than managing outliers through sparse storage or asymmetric groupings, QuaRot and RotateKV eliminate activation outliers mathematically via orthogonal matrix rotations.

• Randomized Walsh-Hadamard Transform (WHT): The architecture applies orthogonal Walsh-Hadamard matrices to the hidden states and projection weights. Because orthogonal transformations preserve vector inner products (H^T H = I), the model's output remains identical.
• Outlier Invariance: Rotating activations spreads concentrated outlier energy evenly across all hidden dimensions, eliminating extreme peaks.
• End-to-End 4-Bit Execution: With outliers flattened, both weights, activations, and KV cache elements are quantized to uniform 4-bit formats (W4A4KV4) without channel grouping or precision fallbacks. Online Fast Walsh-Hadamard Transforms (FWHT) are fused into projection kernels to minimize runtime latency.

Serving Economics and Latency Trade-Offs

Deploying KV cache quantization involves balancing memory capacity gains against kernel computation overheads:

• Time-to-First-Token (TTFT / Prefill Phase): Prefill is compute-bound. Quantizing the KV cache during prefill requires scaling factor calculation and bit-packing operations, adding slight prefill latency unless fused into the attention projection kernel.
• Inter-Token Latency (ITL / Decode Phase): Autoregressive decoding is memory-bandwidth bound. Reducing the byte transfer volume per token directly decreases memory read times. For batch sizes of 32 or larger, 4-bit and 8-bit caches consistently reduce ITL compared to FP16.
• Maximum Concurrency: Halving KV cache footprint from FP16 to FP8 doubles the maximum concurrent token capacity within GPU memory, delaying request queuing under heavy traffic. Shifting to 4-bit representations quadruples available capacity.

Practical Implementation Matrix

• Native Hopper/Blackwell Deployments: Deploy FP8 E4M3 KV cache (--kv-cache-dtype fp8 in vLLM / SGLang). It delivers a 2x capacity increase with native Tensor Core attention support and near-zero accuracy loss.
• Ampere/Ada GPU Fleets: Deploy INT8 per-token quantization or 4-bit group-wise quantization with group size 64 for a balance of memory reduction and kernel compatibility.
• Extreme Context / Edge Runtimes: Deploy KIVI (4-bit or 2-bit with FP16 residual window) or QuaRot W4A4KV4 when serving 100k+ sequence lengths on memory-constrained hardware where memory footprint overrides kernel dequantization costs.

Update (Oct 19, 2026)

• FlashInfer is a customizable attention engine for LLM serving that is integrated into serving frameworks such as SGLang and vLLM (Ye et al., 2025).

• Capacity math: an 8-bit KV cache uses half the memory of a 16-bit cache, so roughly twice as many tokens or concurrent sequences fit in the same GPU memory, and a 4-bit cache fits close to four times as many. vLLM documents its FP8 KV cache options in its quantized KV cache guide.

Sources

• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al., ICML 2024 / arXiv:2402.02750)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (Hooper et al., NeurIPS 2024 / arXiv:2401.18079)
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (Ashkboos et al., NeurIPS 2024 / arXiv:2404.00456)
• RotateKV: Accurate and Robust 2-Bit KV Cache Quantization (IJCAI 2025)
• The State of FP8 KV-Cache and Attention Quantization in vLLM (vLLM Project Technical Report)
• vLLM GitHub Repository
• FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving