Quantized KV Caches in Production: FP8 vs. INT8 vs. INT4 Architecture, Kernel Backends, and Serving Economics
In modern large language model serving, memory capacity and memory bandwidth are the two primary bottlenecks governing inference economics. While static model weights occupy a fixed footprint in GPU High Bandwidth Memory (HBM), the Key-Value (KV) cache grows dynamically with batch size and sequence length. In workloads with 32,000 to 128,000 token context windows, the KV cache quickly overtakes weight memory, consuming up to 70% of total VRAM and capping concurrency. During autoregressive gener
1 min