Hierarchical KV Cache Offloading in Production: Multi-Tier Storage Across HBM, DRAM, NVMe, and Remote Pools

Serving large language models at context lengths of 32,000 to 1 million tokens exposes a hard physical constraint: GPU High-Bandwidth Memory (HBM) capacity. While modern accelerator compute scales efficiently across tensor-parallel and pipeline-parallel topologies, the memory footprint of Key-Value (KV) attention states scales linearly with context length, batch size, number of layers, and hidden dimensions.

On an 80GB NVIDIA H100 GPU running Llama 3 70B in FP16 precision, the model weights alone occupy approximately 140GB across a two-GPU tensor-parallel group. Storing the KV cache for a single 128,000-token sequence requires roughly 80GB of memory. Under standard monolithic serving architectures, a single long-context request consumes the entire spare memory pool of an H100 pair, reducing concurrent serving capacity to near zero.

To overcome the capacity ceiling of GPU HBM without abandoning long-context throughput, production inference systems are shifting to hierarchical KV cache offloading. Modern frameworks such as LMCache, Mooncake, and SGLang distribute KV tensors across a multi-tier memory hierarchy spanning GPU HBM, host system DRAM, local NVMe solid-state drives, and remote RDMA storage pools.

Hierarchical KV Cache Architecture across HBM, DRAM, and NVMe

The Four-Tier KV Storage Hierarchy

Hierarchical KV caching models memory as a tiered pyramid, trading access latency and bandwidth for orders-of-magnitude larger capacity at lower capital cost.

+-------------------------------------------------------------------------+
| Tier 0: GPU High-Bandwidth Memory (HBM3 / HBM3e)                        |
| Bandwidth: 3.35 TB/s - 4.8 TB/s | Latency: < 1 microsecond             |
| Capacity: 80 GB - 144 GB per accelerator                                |
| Role: Active working set for active decoding step and attention kernels |
+-------------------------------------------------------------------------+
                                    |
                    PCIe Gen5 x16 (64 GB/s bi-dir) / NVLink-C2C
                                    v
+-------------------------------------------------------------------------+
| Tier 1: Host System DRAM (DDR5 / CXL 2.0/3.0 Pools)                     |
| Bandwidth: 100 - 400 GB/s (Local) | Latency: 50 - 100 nanoseconds       |
| Capacity: 512 GB - 2 TB per dual-socket host                            |
| Role: Hot prefix buffer, eviction staging, zero-copy pinned swap space  |
+-------------------------------------------------------------------------+
                                    |
                    PCIe Gen5 x4 Direct NVMe / GPUDirect Storage (GDS)
                                    v
+-------------------------------------------------------------------------+
| Tier 2: Local NVMe SSD Arrays (PCIe Gen5 U.2 / E3.S)                    |
| Bandwidth: 7 - 14 GB/s per drive (28 - 56 GB/s array) | Latency: 10-30us|
| Capacity: 4 TB - 30 TB per server chassis                               |
| Role: Persistent local document store, warm multi-turn session cache    |
+-------------------------------------------------------------------------+
                                    |
                    RoCEv2 / InfiniBand (400 Gbps - 800 Gbps NICs)
                                    v
+-------------------------------------------------------------------------+
| Tier 3: Distributed Remote Memory & Object Stores (Mooncake / S3)       |
| Bandwidth: 50 - 100 GB/s network aggregated | Latency: 50us - 20ms      |
| Capacity: 100 TB - Petabytes cluster-wide                               |
| Role: Cross-node prefix sharing, disaggregated prefill-to-decode handoff|
+-------------------------------------------------------------------------+

Tier 0: GPU High-Bandwidth Memory (HBM)

GPU HBM (such as HBM3 on H100 with 3.35 TB/s bandwidth, or HBM3e on H200 with 4.8 TB/s) is the only tier fast enough to feed flash-attention computation during generation without throttling decode throughput. HBM holds the active attention working set for active batch sequences. When memory pressure exceeds high-water mark thresholds (typically 85-90% allocation), eviction engines transfer inactive blocks down the hierarchy.

Tier 1: Host System DRAM

Modern dual-socket AI servers feature 1TB to 2TB of DDR5 host system memory delivering 300 to 400 GB/s of local memory bandwidth. Connected via PCIe Gen5 x16 slots (yielding 64 GB/s unidirectional bandwidth per GPU), host DRAM acts as a primary buffer. By allocating page-locked (pinned) memory on the host, inference engines execute asynchronous DMA transfers between HBM and host memory via CUDA streams without blocking active GPU execution.

Tier 2: Local NVMe Solid-State Storage

High-density enterprise PCIe Gen5 NVMe drives provide sequential read throughput up to 14 GB/s per drive. A four-drive NVMe array provides over 50 GB/s of sustained sequential throughput and up to 30TB of persistent cache capacity per server. Using NVIDIA GPUDirect Storage (GDS) via the cuFile API, NVMe controllers transfer KV blocks directly to GPU HBM over the PCIe bus, bypassing host CPU bouncing and reducing memory copy overhead.

Tier 3: Distributed Remote Memory Pools

In multi-node disaggregated clusters, nodes share KV caches over 400 Gbps or 800 Gbps RoCEv2 and InfiniBand networks. Systems like Mooncake Store implement peer-to-peer RDMA transport to move KV chunks directly between the DRAM/NVMe of prefill nodes and the HBM of decode nodes.

The Economics of Prefill Recomputation vs. Storage Retrieval

The engineering justification for hierarchical offloading rests on an arithmetic reality: reading precomputed KV states from secondary storage is drastically faster than running dense prefill compute for long prompts.

For a transformer model with hidden size $H$ , number of layers $L$ , sequence length $S$ , and floating-point precision $P$ bytes:

$\text{KV Size (bytes)} = 2 \times L \times H \times S \times P$

For a 70B parameter model with 80 layers, hidden dimension 8192 (or equivalent key/value projection size under grouped-query attention), and 128,000 input tokens in FP16 ( $P=2$ ):

Compute Cost: Computing the prefill forward pass on 128,000 tokens requires approximately $2 \times 70 \times 10^9 \times 128,000 \approx 1.79 \times 10^{16}$ floating-point operations. On an 8-GPU H100 cluster running at practical MFU (Model FLOPs Utilization) of 45% (effective cluster throughput of roughly 3,500 TFLOPs FP16/BF16), computing prefill takes approximately 5.1 seconds.
Host DRAM Retrieval: Transferring the ~80GB KV tensor across an aggregate PCIe Gen5 x16 host link at 50 GB/s effective throughput takes 1.6 seconds.
Local NVMe Retrieval: Streaming the ~80GB KV tensor from a four-drive Gen5 NVMe array at 45 GB/s sustained throughput takes 1.77 seconds.

When using 8-bit or 4-bit KV quantization, the data transfer size shrinks to 40GB or 20GB, reducing NVMe retrieval time to less than 500 milliseconds. Hierarchical retrieval cuts Time-To-First-Token (TTFT) by 70% to 90% compared to raw prefill computation, while freeing compute-heavy GPUs to execute active token generation.

System Architectures: LMCache and Mooncake

Two prominent architectures illustrate modern production patterns for hierarchical KV management.

LMCache: Plug-and-Play Serialization and Storage Connectors

LMCache acts as an external tensor storage layer that integrates into inference engines like vLLM and SGLang.

+------------------------------------------------------------------------+
|                          LLM Inference Engine                          |
|         (vLLM / SGLang / TensorRT-LLM with Radix Prefix Tree)          |
+------------------------------------------------------------------------+
                                    |
                   LMCache Adapter (Tensor Interceptor)
                                    |
          +-------------------------+-------------------------+
          |                                                   |
          v                                                   v
+-----------------------+                           +-------------------+
|   L1 Metadata Engine  |                           |  L2 SerDe Engine  |
| - Chunk Hashing (SHA) |                           | - Tensor Packing  |
| - Prefix Indexing     |                           | - Quant (FP8/INT4)|
| - Cache Eviction LRU  |                           | - Delta Encoding  |
+-----------------------+                           +-------------------+
          |                                                   |
          +-------------------------+-------------------------+
                                    |
                         Pluggable Storage Engine
                                    |
       +----------------------------+----------------------------+
       |                            |                            |
       v                            v                            v
[ Host DRAM Pool ]          [ Local NVMe / GDS ]        [ Redis / S3 / RDMA ]

LMCache segments incoming prompts into fixed-size token chunks (typically 256 or 512 tokens). Each chunk is deterministically hashed using cryptographic hashes of the token sequence and parent prefix hashes.

When a prompt matches a known prefix tree path:

The L1 metadata engine identifies cached token chunks across storage tiers.
The L2 Serialization/Deserialization (SerDe) module issues parallel asynchronous fetch requests to the lowest available latency tier.
Retrieved tensors are deserialized, optionally dequantized, and loaded into pre-allocated vLLM PagedAttention blocks before generation begins.

Mooncake: Disaggregated KV-Centric Architecture

Developed by Moonshot AI to power the Kimi service, Mooncake separates inference into distinct prefill and decode clusters connected by a unified distributed memory plane.

Instead of discarding prefill KV states after computing prompt attention, Mooncake's prefill instances stream KV chunks over RDMA into a distributed Mooncake Store running across the cluster's host DRAM and NVMe drives. When a decode instance receives the request, it pulls the required KV chunks directly from the nearest Mooncake Store shard.

In production benchmarks published at FAST 2025, Mooncake achieved up to 525% throughput improvements in long-context workloads while maintaining strict Service Level Objectives (SLOs), handling 75% higher request volume under real production traffic.

KV Compression and Streaming Across Tiers

Transferring uncompressed FP16 KV states across PCIe and network links introduces bus contention. Modern systems apply lossy and lossless compression pipelines before offloading tensors to secondary storage.

Layer-Wise Quantization

KV tensors exhibit structured outlier distributions across channel dimensions. Quantizing KV states to FP8 (E4M3 or E5M2) or INT4 reduces storage footprint by 50% to 75% with negligible perplexity degradation. Systems like KIVI implement asymmetric per-channel key quantization and per-token value quantization, enabling 2-bit and 4-bit KV storage that quadruples the effective throughput of PCIe and NVMe transfer links.

Delta Encoding and Information Theoretic Compression

Research in CacheGen demonstrates that attention keys and values across adjacent layers and positions exhibit high spatial correlation. CacheGen applies dynamic delta encoding:

Keys and values are anchored against reference vectors.
Residual differences are encoded using custom arithmetic coders.
Tensors achieve up to 4.3x compression ratios over standard float representations.

By compressing KV blocks prior to host or network transmission, transfer latency drops below the decompression execution budget on the host CPU or tensor cores.

Data Movement Mechanics: Pinned Buffers and GPUDirect Storage

Achieving line-rate transfers between storage tiers requires eliminating CPU intermediate copy buffers.

Standard File I/O Path:
[ NVMe Disk ] ---> [ OS Kernel Page Cache ] ---> [ Host User Buffer ] ---> [ GPU HBM ]
(High CPU utilization, 3x memory copies, context switches)

GPUDirect Storage (GDS) Path:
[ NVMe Disk ] ------------------- DMA over PCIe -------------------------> [ GPU HBM ]
(Zero CPU overhead, zero bounce buffers, sub-microsecond driver overhead)

Pinned Host Memory Pools: When swapping between HBM and host DRAM, inference servers pre-allocate large pinned (page-locked) memory buffers via cudaMallocHost(). Pinned memory allows the GPU Direct Memory Access (DMA) engine to read and write system RAM without operating system paging interrupts.
GPUDirect Storage (cuFile): For NVMe offloading, cuFile establishes direct DMA mappings between NVMe controller memory registers and GPU virtual addresses. The host CPU initializes the I/O descriptor, but data flows directly across PCIe root complexes at hardware wire speeds.
Pipelined Asynchronous Prefetching: In multi-turn agentic workflows, systems initiate background prefetching of anticipated session prefixes while current turns are waiting on external tool execution or network I/O, ensuring zero-latency cache hits when the model resumes.

Production Pitfalls and Failure Modes

Operating hierarchical KV caches in high-concurrency production environments introduces several engineering challenges:

PCIe Bus Congestion: On servers sharing PCIe lanes between GPUs, NICs, and NVMe drives, aggressive KV swapping can saturate PCIe root complexes, throttling inter-GPU tensor-parallel all-reduce operations. Production setups isolate NVMe drives on dedicated PCIe switches or utilize separate CXL root ports.
Cache Fragmentation and Block Invalidation: Dynamic prompts with variable system headers or timestamp insertions break prefix matching trees. Systems require standardized prompt framing that isolates dynamic variables to the end of input sequences, maximizing prefix reuse.
Pinned Memory Exhaustion: Over-allocating host DRAM for page-locked KV buffers can trigger operating system kernel out-of-memory (OOM) faults. Memory managers must maintain strict static quotas with fallback to unpinned eviction staging.
Stale State in Distributed Clusters: In multi-node setups, node failures or network partitions can leave orphaned KV chunks in remote storage. Distributed stores must enforce time-to-live (TTL) leases and cryptographic cache verification to prevent silent state corruption.

Hierarchical KV caching transforms long-context LLM serving from a memory-bound bottleneck into an engineered storage hierarchy problem, enabling high-concurrency enterprise workloads at sustainable infrastructure costs.