Storage

Hierarchical KV Cache Offloading in Production: Multi-Tier Storage Across HBM, DRAM, NVMe, and Remote Pools

Serving large language models at context lengths of 32,000 to 1 million tokens exposes a hard physical constraint: GPU High-Bandwidth Memory (HBM) capacity. While modern accelerator compute scales efficiently across tensor-parallel and pipeline-parallel topologies, the memory footprint of Key-Value (KV) attention states scales linearly with context length, batch size, number of layers, and hidden dimensions. On an 80GB NVIDIA H100 GPU running Llama 3 70B in FP16 precision, the model weights alo