In modern large language model serving, memory capacity and memory bandwidth are the two primary bottlenecks governing inference economics. While static model weights occupy a fixed footprint in GPU High Bandwidth Memory (HBM), the Key-Value (KV) cache grows dynamically with batch size and sequence length. In workloads with 32,000 to 128,000 token context windows, the KV cache quickly overtakes weight memory, consuming up to 70% of total VRAM and capping concurrency.
During autoregressive generation, decoding is fundamentally memory-bandwidth bound. Compute cores spend significant cycle time waiting for KV tensors to load from HBM into on-chip SRAM and registers. Quantizing the KV cache from 16-bit precision (FP16 or BF16) down to 8-bit (FP8, INT8) or 4-bit formats addresses both constraints simultaneously: it halves or quarters cache memory requirements and cuts the data volume transferred across the memory bus during each generation step.
The Mathematics of KV Cache Growth
For a standard Transformer model with multi-head attention (MHA) or grouped-query attention (GQA), the memory consumed by the KV cache per token across all layers is calculated as:
Memory per token = 2 * n_layers * n_kv_heads * d_head * bytes_per_element
In 16-bit precision (2 bytes per element), a model like Llama 3 70B (80 layers, 8 KV heads, head dimension 128) requires 160 KB of KV cache per token. A single request processing a 64k-token document consumes roughly 10 GB of HBM just to hold context states. A concurrent batch of 8 such requests requires 80 GB, exhausting the memory of an entire NVIDIA H100 GPU before accounting for model parameters or runtime scratchpads.
With Grouped-Query Attention, models share key and value heads across multiple query heads to reduce footprint. However, as enterprise applications move toward agentic workflows, long documents, and multi-turn tool interactions, GQA alone is insufficient to prevent memory exhaustion during peak traffic.
Numerical Formats: FP8 vs. INT8 vs. INT4
Production inference frameworks support multiple numerical formats for KV cache quantization, each carrying specific mathematical and implementation trade-offs:
- FP8 (E4M3 vs. E5M2): The IEEE and OCP 8-bit floating-point specifications define two primary variants: E4M3 (1 sign bit, 4 exponent bits, 3 mantissa bits) and E5M2 (1 sign bit, 5 exponent bits, 2 mantissa bits). In KV cache serving, E4M3 is universally preferred over E5M2. Because layer normalization constrains attention inputs to bounded numerical ranges, the wider dynamic range of E5M2 is unnecessary. E4M3 provides higher precision (8 discrete levels per power of two instead of 4), minimizing quantization noise in attention dot-products.
- INT8 (Uniform Integer Quantization): INT8 maps floating-point numbers linearly to signed 8-bit integers using a scaling factor:
q = round(x / s). While INT8 provides uniform resolution across the dynamic range, attention Keys often develop channel-specific outliers across long sequences. Without fine-grained per-channel scaling, uniform INT8 can clip salient features, degrading attention accuracy. - INT4 and Sub-4-Bit Compression: Pushing KV cache quantization to 4 bits or 2 bits requires asymmetric treatment. As demonstrated in the KIVI framework, Key caches exhibit prominent outlier channels along the hidden dimension, whereas Value caches vary predominantly across tokens. KIVI applies per-channel quantization to Keys and per-token quantization to Values, maintaining output quality down to 2-bit representations without fine-tuning. Rotation techniques like QuaRot apply randomized orthogonal Walsh-Hadamard transforms to activations, dispersing outlier peaks across all channels before uniform 4-bit quantization.
Kernel Backends and Execution Mechanics
The operational gains of KV cache quantization depend heavily on the underlying GPU architecture and attention kernel implementation:
- Native FP8 Tensor Core Math (NVIDIA Hopper / Blackwell / Ada): On SM90 (H100) and SM89 (L40S) architectures, Tensor Cores support native FP8 matrix multiplications. Kernels in libraries such as FlashInfer and FlashAttention-3 execute the scaled dot-product attention directly on FP8 Key and Value buffers. Queries are quantized on the fly, allowing matrix multiplications (Q * K^T and Attention_Score * V) to run directly on FP8 hardware pipelines without intermediate dequantization.
- Dequantization on Load (NVIDIA Ampere / SM80): On older architectures like the A100, Tensor Cores do not support native FP8 arithmetic. Serving engines load packed 8-bit or 4-bit data from HBM to registers, then convert elements back to BF16/FP16 before computing attention. Because decoding is memory-bandwidth bound rather than compute bound, this approach still yields substantial throughput gains by reducing memory traffic, despite the extra register-level arithmetic instructions.
- PagedAttention and Memory Layouts: Inference engines like vLLM and SGLang manage KV cache memory in non-contiguous physical blocks using PagedAttention. When storing FP8 or INT4 tensors, the block allocator manages packed byte layouts and associated scaling metadata (per-tensor, per-head, or per-token scale factors) alongside the page table structures.
Accuracy, Latency, and Serving Economics
Deploying quantized KV caches introduces measurable trade-offs across inference metrics:
- Capacity and Concurrency: Moving from 16-bit to 8-bit KV caches cuts memory requirements in half, effectively doubling the maximum batch size that fits in GPU memory. For 4-bit formats, capacity increases nearly fourfold. This allows serving clusters to handle higher request spikes without queuing or dropping tokens.
- Inter-Token Latency (ITL): In memory-bound generation phases, halving the bytes transferred per token reduces memory bus saturation. Benchmarks on FlashInfer and vLLM show 10% to 25% reductions in median Inter-Token Latency for decode-heavy workloads when operating with large batch sizes and long contexts.
- Retrieval Fidelity: On multi-needle retrieval and long-context reasoning benchmarks (such as Ruler and MRCR) up to 128k context, FP8 E4M3 retains 97% to 99% of full-precision accuracy across standard models including Llama 3 and Qwen. INT4 implementations using Hadamard rotations or per-channel scaling maintain baseline performance on standard language modeling tasks, though uncalibrated 4-bit schemes show degradation in complex multi-hop retrieval.
Implementation Best Practices
For teams configuring production serving pipelines, the following architectural guidelines apply:
- Default to FP8 E4M3 on Modern Hardware: On NVIDIA Hopper, Ada, or Blackwell GPUs, set FP8 E4M3 as the default KV cache format. The combination of native Tensor Core execution and minimal accuracy degradation provides immediate cost efficiency.
- Calibrate Scaling Factors When Necessary: For models exhibiting high activation variance, utilize calibration tools like llm-compressor to compute static per-channel or per-head scaling factors prior to deployment, avoiding runtime dynamic scaling overhead.
- Monitor RoPE Vector Quantization: In architectures with Rotary Position Embeddings, quantizing high-frequency positional components can cause phase drift over long contexts. Several modern implementations retain RoPE position coordinates in unquantized BF16 while compressing content latents to FP8.
Sources
- FlashInfer: Efficient and Customizable Attention Engine for LLM Serving (arXiv:2501.01005)
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (arXiv:2402.02750)
- QuaRot: Outlier-Free 4-Bit LLM Inference via Rotation (arXiv:2404.00456)
- vLLM Quantized KV Cache Documentation
- The State of FP8 KV-Cache and Attention Quantization in vLLM
