GPU Cluster Storage in Production: GPUDirect Storage, NVMe-oF, Parallel File Systems, and Checkpointing Throughput

Training frontier large language models and serving hundred-billion parameter checkpoints places extreme demands on storage subsystems. While compute clusters frequently deploy thousands of GPUs connected via high-bandwidth interconnects like NVLink and InfiniBand, storage architectures often become severe bottlenecks during two critical operational phases: distributed checkpointing and cold-start model weight loading.

A standard 70-billion parameter model in BF16 precision generates approximately 140 GB of state per checkpoint, expanding to over 400 GB when optimizer states (such as FP32 momentum and variance in AdamW) and distributed sharding metadata are preserved. At the 405-billion parameter scale, full cluster checkpoints easily reach multiple terabytes. If storage I/O bandwidth is insufficient, thousands of compute-dense GPUs sit idle during synchronous save operations, degrading overall model training compute efficiency.

Similarly, in multi-node inference serving across tensor-parallel (TP) and pipeline-parallel (PP) ranks, pulling multi-hundred gigabyte weight files over legacy file protocols can lead to minutes of cold-start latency. Resolving these bottlenecks requires moving away from traditional CPU-buffered POSIX I/O toward zero-copy direct memory access (DMA), parallel file systems, and tiered asynchronous persistence fabrics.

The Bipolar I/O Profile of Large Language Models

Storage requirements for LLM infrastructure exhibit a sharp duality between training ingestion and checkpoint lifecycle management:

Training Data Ingestion: Characterized by millions of continuous random reads across tokenized dataset shards (such as Parquet, WebDataset, or custom binary files). This stage demands high random read IOPS, minimal metadata latency, and effective caching layers to prevent GPU starvation.
Distributed Checkpointing: Characterized by bursty, massive sequential writes where all participating GPU ranks dump their local model weights, optimizer tensors, and learning rate scheduler states simultaneously. This stage requires high aggregate sequential write throughput.
Inference Cold Starts: Characterized by high-bandwidth sequential reads during container initialization or worker failover, where models must be rapidly mapped into GPU High-Bandwidth Memory (HBM).

When storage systems are tuned solely for standard enterprise file sharing or general-purpose virtualization, they fail under these synchronized read/write spikes.

Kernel Bypass: CPU Bounce Buffers vs. GPUDirect Storage

The standard Linux POSIX storage path introduces substantial serialization overhead. In a conventional file read or write pipeline, data moves through several intermediate hops:

Data is transferred from the storage device (NVMe drive or network interface card) into the Linux kernel page cache in CPU DRAM via Direct Memory Access.
The operating system copies the data from kernel memory into a user-space application buffer in CPU memory via read() or write() system calls.
The training or inference runtime executes an asynchronous CUDA memory copy (cudaMemcpyAsync) over the PCIe bus to transfer data from CPU host memory into GPU High-Bandwidth Memory (HBM).

This multi-hop pipeline creates severe CPU utilization spikes, forces redundant memory copies across the PCIe root complex, and limits throughput to the memory bandwidth of host CPU channels.

NVIDIA Magnum IO GPUDirect Storage (GDS)

To bypass host CPU bottlenecks, NVIDIA GPUDirect Storage establishes a direct DMA transfer path between local NVMe drives (or remote NVMe-oF network adapters) and GPU memory.

Architecturally, GDS relies on two primary components:

The cuFile Userspace Library: Provides an asynchronous, POSIX-like API (cuFileRead, cuFileWrite, cuFileHandleRegister) that enables applications to direct I/O requests directly to GPU virtual memory addresses.
The nvidia-fs Kernel Module: A lightweight kernel driver that intercepts file operations, maps the physical page addresses of GPU High-Bandwidth Memory, and programs the storage controller or NIC DMA engines to transfer blocks directly across PCIe switches without staging data in CPU RAM.

By eliminating CPU bounce buffers and context-switch overhead, GDS achieves significant performance gains:

Checkpoint Write Latency: On single-node H100 systems equipped with PCIe Gen5 NVMe arrays, GDS reduces the time required to dump a 140 GB BF16 checkpoint from over 4 minutes down to under 45 seconds.
Inference Cold Starts: In serving frameworks like vLLM, loading pre-sharded weights for models such as Llama 3.1 405B over a GDS-accelerated parallel storage fabric cuts cold-start loading time from approximately 18 minutes down to 6 to 10 seconds.

Network Fabrics: NVMe-oF vs. Parallel File Systems

In multi-node GPU clusters, local NVMe storage alone is insufficient for coordinated model training. Compute nodes require shared, high-throughput network storage accessible across hundreds or thousands of GPU workers.

NVMe over Fabrics (NVMe-oF)

NVMe over Fabrics extends the NVMe protocol across remote network fabrics using Remote Direct Memory Access (RDMA) over InfiniBand or RoCEv2 (RDMA over Converged Ethernet). NVMe-oF delivers sub-10-microsecond access latencies and raw block-level throughput approaching the physical line rate of the underlying network adapters (such as 400 Gbps ConnectX-7 or 800 Gbps ConnectX-8 interfaces). However, because NVMe-oF provides raw block storage rather than a shared namespace, multi-node training clusters typically run a distributed or parallel file system on top of the underlying NVMe pool.

Lustre

Lustre is the established open-source parallel file system in high-performance computing (HPC) and large-scale AI supercomputing. Lustre decouples metadata operations from data storage through dedicated server roles:

Metadata Servers (MDS) and Targets (MDT): Manage directory hierarchies, file permissions, and inode metadata.
Object Storage Servers (OSS) and Targets (OST): Store the raw file data chunks striped across multiple underlying block storage arrays.

Lustre excels at massive sequential I/O for multi-terabyte checkpoints when client-side file striping (lfs setstripe) is configured across dozens of OSTs. However, managing millions of small dataset files can create heavy contention on the MDS unless distributed namespace (DNE) features and large MDT caches are properly provisioned.

WekaFS

WEKA is a software-defined, distributed parallel file system built specifically for flash and NVMe architectures. Unlike traditional kernel-based file systems, WekaFS runs in user space with dedicated CPU core pinning, using standard POSIX semantics and full GPUDirect Storage integration.

WekaFS distributes metadata and data blocks evenly across all participating NVMe nodes using distributed erasure coding (such as N+2 or N+4 schemes) and custom DPDK-based networking. This allows it to handle both high-IOPS small-file random reads during dataset ingestion and sustained line-rate sequential writes during distributed checkpointing without requiring separate metadata server tuning.

JuiceFS

JuiceFS is an open-source, POSIX-compliant distributed file system designed to bridge cloud object storage (such as Amazon S3, Google Cloud Storage, or MinIO) with high-performance GPU compute.

JuiceFS separates data storage from metadata management:

Metadata Engine: High-performance transactional key-value stores (such as Redis, TiKV, or MySQL) handle all POSIX metadata, file lookups, and chunk indexing.
Data Storage: Raw file chunks are compressed, encrypted, and stored in standard object storage.
Local Flash Caching: The JuiceFS client running on GPU worker nodes automatically caches frequently read dataset blocks in local NVMe drives and kernel page cache.

For LLM training, JuiceFS provides virtually limitless capacity backed by cost-effective object storage, while achieving local-NVMe read throughput (often exceeding 10 GB/s per compute node on warm caches) during training data iteration.

Checkpoint Optimization in Distributed Frameworks

Beyond storage hardware and file systems, the software architecture used to serialize model state plays a fundamental role in cluster stability.

Synchronous vs. Asynchronous Checkpoint Staging

In a naive synchronous checkpoint implementation, training loops execute a blocking write call:

# Synchronous blocking write (pauses GPU compute)
torch.save(model.state_dict(), "/shared/storage/checkpoints/step_1000.pt")

During this call, all GPU tensor cores halt execution while data transfers over the network. If the storage target experiences transient latency spikes or network congestion, compute utilization collapses.

In an asynchronous staging architecture, GPU workers dump their local state into high-speed local NVMe flash or pinned CPU host memory in seconds, allowing the training loop to resume immediately:

Step N Complete: Model weights and optimizer states are dumped locally to node-attached NVMe via GDS or high-speed local RAM.
Compute Resumes: GPU training immediately starts on Step N+1.
Background Worker: An asynchronous daemon or background thread streams the staged checkpoint from local scratch to the shared Tier 1 parallel file system or remote object store.

PyTorch Distributed Checkpointing (DCP)

Modern distributed training libraries (including Megatron-LM, DeepSpeed, and PyTorch FSDP) utilize PyTorch Distributed Checkpointing (torch.distributed.checkpoint).

Traditional torch.save operations often require gathering full model tensors to Rank 0, creating massive memory overhead and network bottlenecks. In contrast, PyTorch DCP writes sharded state dicts concurrently from all distributed ranks:

Each GPU rank writes only its local parameter shards and optimizer slices directly into assigned storage targets.
Checkpoints are written in parallel without requiring all-gather communication or un-sharding during save time.
Resumption supports dynamic resharding, allowing training to restart on a different GPU cluster topology (for example, switching from 128 GPUs to 64 GPUs) without manual offline conversion.

Production Storage Tiering Architecture

A production-grade storage topology for large language model development and serving typically adopts a three-tier architecture:

+-------------------------------------------------------------------+
| Tier 0: Ephemeral Local NVMe Scratch                              |
| Direct-attached Gen5 NVMe / GDS-enabled / Local KV Cache Spillover|
+---------------------------------+---------------------------------+
                                  | Asynchronous Staging
+---------------------------------v---------------------------------+
| Tier 1: High-Performance Shared Parallel File System               |
| WekaFS / Lustre / GPFS / InfiniBand & RoCEv2 Network Fabric       |
| Active Training Corpora / DCP Checkpoints / Shared Serving Models |
+---------------------------------+---------------------------------+
                                  | Lifecycle Archival
+---------------------------------v---------------------------------+
| Tier 2: Cloud Object Storage                                      |
| Amazon S3 / Google Cloud Storage / Ceph Object Gateway            |
| Golden Model Registries / Raw Pre-Training Datasets / Cold Backups|
+-------------------------------------------------------------------+

1. Tier 0: Ephemeral Local Scratch

Medium: Direct-attached NVMe SSDs (PCIe Gen4/Gen5) installed directly on GPU compute nodes.
Function: Local dataset shuffle caches, asynchronous checkpoint staging buffers, and ephemeral KV cache paging for long-context inference serving.
Access Path: GPUDirect Storage direct to GPU HBM.

2. Tier 1: Shared Parallel File System

Medium: Networked all-flash NVMe arrays running WekaFS, Lustre, or high-throughput NFS over RDMA.
Function: Hosting tokenized training datasets, active training run checkpoints, and shared model weights accessible across all compute nodes.
Access Path: POSIX client with multi-NIC RDMA trunking.

3. Tier 2: Object Storage Archive

Medium: Cloud object storage (S3, GCS) or on-premises Ceph clusters.
Function: Long-term archival of finalized model weights, cold checkpoint snapshots, raw raw-text training data, and compliance audits.
Access Path: S3 API via asynchronous data pipelines.

Architectural Trade-Offs and Failure Modes

Operating high-performance storage at GPU scale requires proactive management of several subtle failure modes:

Fabric Contention: When storage traffic shares the same physical network interfaces or switches as inter-GPU collective communication (NCCL All-Reduce / Reduce-Scatter), bursty checkpoint writes can introduce packet loss and tail latency in training steps. Dedicated storage network adapters (isolated rail-optimized networks) prevent cross-traffic degradation.
PCIe Root Complex Congestion: On dual-socket CPU host nodes, placing NVMe drives and GPUs on different NUMA nodes forces transfers across inter-socket UPI/QPI links, degrading GDS bandwidth by up to 40%. Storage NICs and NVMe devices must be co-located on the same PCIe switches as their target GPUs.
Metadata Bottlenecks: Storing uncompressed raw dataset files as millions of small text or image files creates severe metadata locking on storage controllers. Production pipelines must pack training data into monolithic binary containers (such as WebDataset .tar archives or Parquet files with 100MB+ row groups) to maintain streaming read efficiency.

Designing storage infrastructure around GPU memory direct access, asynchronous checkpointing pipelines, and clear tier boundaries ensures that expensive compute clusters operate at maximum training and inference efficiency.