Fast Model Weight Loading in Production: Safetensors, Tensorizer, and Direct GPU Deserialization

In modern large language model inference clusters, cold start latency is rarely bounded by GPU compute allocation. Instead, the operational bottleneck centers on storage I/O and weight deserialization. As foundation models scale from 70 billion to 405 billion parameters, raw weight footprints range from 140 GB to over 800 GB in standard 16-bit precision. On naive serving stacks, deserializing these parameters across multi-GPU nodes takes anywhere from several minutes to upwards of twenty minutes, leaving thousands of dollars in accelerator hardware sitting idle.

Addressing cold start latency requires rethinking the journey model weights take from persistent storage into GPU High Bandwidth Memory (HBM). Traditional deserialization paths that rely on host CPU memory bounce buffers, unaligned disk reads, and Python runtime overhead are increasingly replaced by zero-copy memory mapping, streaming serialization protocols, and direct kernel-bypass DMA transfers.

The Bottlenecks of Traditional Weight Loading

The standard model loading pipeline in early deep learning frameworks was designed for small convolutional networks, not distributed parameter matrices spanning tens of gigabytes per shard.

Three architectural bottlenecks contribute to slow weight loading in production:

1. Python Pickle and CPU-Side Deserialization Overhead

Historically, PyTorch checkpoints relied on the torch.save and torch.load interfaces built on top of Python pickle. Unpickling large state dictionaries introduces substantial CPU overhead. The Python interpreter sequentially parses dictionary objects, allocates individual Python object wrappers for every tensor, and runs into Python Global Interpreter Lock (GIL) contention. Furthermore, pickle allows arbitrary code execution during deserialization, creating severe security vulnerabilities in multi-tenant environments.

2. The Host Memory Bounce Buffer Cascade

In a naive loading path, weights follow an indirect multi-hop memory route:

1. Persistent Storage (NVMe SSD or Network Block Device)
2. Linux Kernel Page Cache
3. Host RAM (User Space CPU Memory)
4. PyTorch Caching Allocator
5. Pinned Host Memory (cudaHostAlloc)
6. PCIe Host-to-Device (H2D) Transfer (cudaMemcpyAsync)
7. GPU Device Memory (HBM)

This multi-stage cascade requires allocating duplicate memory buffers in host RAM. For an 8-GPU node loading a 140 GB model, the host system often requires 200 GB to 300 GB of free system memory simply to stage parameters before pushing them across the PCIe bus. If host RAM is constrained, the operating system triggers aggressive swapping or page thrashing, crashing inference containers before execution begins.

3. Scattered Small I/O Syscalls

A typical transformer checkpoint contains thousands of individual weight tensors across multi-head attention layers, projection matrices, normalization layers, and feed-forward networks. Reading each tensor with individual POSIX read() system calls results in scattered, non-sequential disk access. NVMe storage controllers achieve maximum throughput on large, sequential block transfers; issuing thousands of granular I/O requests throttles read throughput to a fraction of the hardware ceiling.

Serialization Formats Compared

Modern inference runtimes like vLLM and SGLang have decoupled model storage from legacy Python serialization. Three dominant formats govern high-performance weight storage:

Hugging Face Safetensors

Introduced by Hugging Face, Safetensors is a lightweight, secure storage format specifically structured for zero-copy memory mapping.

A Safetensors file consists of two primary components:

• Header Length and JSON Metadata: An initial 8-byte unsigned little-endian integer specifying the byte size of the JSON header, followed by UTF-8 JSON metadata. The header catalogs tensor names, data types, shapes, and absolute byte offsets (data_offsets) within the binary payload.
• Flat Contiguous Binary Payload: Raw tensor bytes stored contiguously in memory, aligned to avoid padding overhead.

By enforcing a simple contiguous layout, Safetensors allows the host operating system to invoke mmap(), mapping the file directly into process virtual memory without parsing Python bytecode. However, standard Safetensors loaders in PyTorch still instantiate each tensor on the CPU before copying it to the target CUDA device.

CoreWeave Tensorizer

Developed by CoreWeave, Tensorizer is a high-speed serialization library optimized for cloud and containerized environments where models reside in remote object storage (Amazon S3, Google Cloud Storage, or HTTP endpoints).

Rather than requiring the host container to download full model shards to local disk before initialization, Tensorizer streams weights directly over network sockets into GPU memory. Key architectural features include:

• Direct-to-VRAM Deserialization: Streamed byte buffers are deserialized straight into target GPU memory pointers, bypassing local disk persistence and host memory buffers.
• Concurrent Chunked Readers: Tensorizer uses multi-threaded HTTP/S3 range requests (num_readers) to saturate available network bandwidth.
• Inline Cryptography: Supports authenticated symmetric encryption (ChaCha20-Poly1305), allowing proprietary weights to be decrypted on the fly as they arrive over the wire.

GGUF (GPT-Generated Unified Format)

Created for llama.cpp and local inference stacks, GGUF stores quantized weights and architecture metadata in a single binary container. GGUF mandates 32-byte alignment across all tensor data, ensuring that memory-mapped file buffers align with SIMD and GPU vectorized load instructions without manual pointer realignment.

Bypassing Host RAM: fastsafetensors and GPUDirect Storage

To eliminate the CPU bounce buffer bottleneck entirely when loading Safetensors checkpoints from local NVMe or high-speed distributed filesystems, researchers and infrastructure teams developed direct-to-GPU loading engines.

Aggregated Tensor Deserialization

As detailed in the research paper Speeding up Model Loading with fastsafetensors (Yoshimura et al., IEEE CLOUD 2025), traditional deserialization processes each tensor sequentially: metadata lookup, host buffer allocation, file read, and device copy.

The fastsafetensors library restructures this execution model:

1. Aggregated Direct I/O: The loader identifies large contiguous byte ranges containing multiple consecutive tensors. It allocates a single large staging buffer directly in GPU VRAM (for example, 10 GB) and transfers the entire block in one high-bandwidth I/O operation.
2. GPU-Offloaded Preprocessing: Once the aggregated binary chunk resides in GPU memory, lightweight CUDA kernels slice individual tensor pointers, handle tensor parallel sharding, and perform data type casting (such as BF16 to FP16 conversion) in parallel across thousands of GPU cores.
3. Memory Recycling: The temporary GPU staging buffer is freed immediately after tensor extraction or reused as the initial memory allocation for the runtime KV cache pool.

Microbenchmarks published in the IEEE study show that fastsafetensors achieves a 4.8x to 7.5x speedup over standard Safetensors loaders on Llama and Falcon models, scaling NVMe read throughput up to 26.4 GB/s across four GPUs.

NVIDIA GPUDirect Storage (GDS)

For deployments with local NVMe arrays or high-performance distributed filesystems (such as Amazon FSx for Lustre), NVIDIA GPUDirect Storage (GDS) bypasses the CPU and Linux kernel page cache entirely.

GDS uses the cuFile API to establish direct Direct Memory Access (DMA) transfers between the NVMe storage controller and GPU HBM through PCIe switches. As documented in AWS engineering benchmarks, pairing GDS with fastsafetensors eliminates CPU context switching and socket bounce buffers, loading full 8-GPU Llama 3.1 70B checkpoints in under two seconds of pure storage transfer time.

Distributed Loading Topologies Across Multi-GPU Nodes

When orchestrating multi-GPU inference instances (such as Tensor Parallelism across 8x H100 or 8x A100 systems), infrastructure teams must choose between two primary weight loading topologies:

1. Parallel Rank Loading

In parallel rank loading, all GPU worker processes independently read their assigned tensor shards from storage concurrently.

• Advantage: Fully utilizes aggregate storage bandwidth across multi-lane PCIe root complexes or distributed filesystems.
• Constraint: Requires storage backends capable of handling high concurrent IOPS. On standard cloud volumes (such as AWS EBS gp3 capped at 1,000 MB/s), concurrent multi-rank reads saturate the I/O ceiling immediately, degrading performance unless provisioned with high-throughput io2 or local NVMe instance store volumes.

2. Broadcast via High-Speed Interconnects (Rank-0 Ingest)

In environments where external network storage bandwidth is limited but internal GPU interconnects are massive (such as 900 GB/s bidirectional NVLink), the cluster designates Rank 0 as the ingest worker.

• Rank 0 loads the primary model weights into its local VRAM buffer.
• Rank 0 broadcasts relevant tensor slices to sibling GPUs across the NVLink fabric using NCCL collective primitives (ncclBcast or tree-based scatter operations).
• This topology isolates storage I/O to a single pipeline while offloading distributed fan-out to ultra-high-bandwidth accelerator fabrics.

Architectural Trade-Offs and Best Practices

Different serving workloads dictate distinct loading strategies:

• Standard Safetensors (mmap): Best for local SSD and EBS deployments where standard compatibility across multiple frameworks is required. Retains moderate host RAM page cache footprints and acts as the baseline for reliability.
• CoreWeave Tensorizer: Best for serverless and ephemeral worker environments where models are stored in S3 or HTTP object stores. Keeps host RAM footprints under 2 GB and delivers 3x to 5x faster network initialization by eliminating local disk staging.
• fastsafetensors with GDS: Best for dedicated high-throughput multi-GPU clusters backed by local NVMe arrays or Lustre filesystems. Delivers 5x to 8x speedups, near-zero host CPU utilization, and sub-second storage-to-VRAM transfers.
• Run:ai Model Streamer: Best for large-scale Kubernetes inference fleets streaming weights across high-throughput S3 or parallel NFS mounts.

To achieve sub-five-second cold starts in production LLM deployments:

1. Pre-shard Weights by Tensor Parallel Degree: Avoid runtime tensor slicing across ranks by storing checkpoints pre-sharded into exact rank distributions (tp_size=8).
2. Standardize on Zero-Copy Serialization: Deprecate legacy PyTorch .pt/.bin checkpoints in favor of Safetensors or Tensorizer binaries to eliminate pickle security risks and CPU deserialization overhead.
3. Align Storage Bandwidth with GPU Count: When using local NVMe instance storage, ensure parallel readers utilize asynchronous I/O (io_uring or cuFile) to prevent single-threaded kernel bottlenecks.
4. Pipeline Layer Allocation with Inference Warmup: Overlap weight transfers of later transformer layers with engine initialization and CUDA graph capture of initial layers.

Sources

• Hugging Face: Safetensors Serialization Repository
• ArXiv: Speeding up Model Loading with fastsafetensors (Yoshimura et al., IEEE CLOUD 2025)
• Foundation Model Stack: fastsafetensors Open-Source Implementation
• CoreWeave: Decrease PyTorch Model Load Times with CoreWeave's Tensorizer
• CoreWeave: Tensorizer Serialization and Deserialization Library
• vLLM Documentation: Loading Models with CoreWeave's Tensorizer and fastsafetensors
• NVIDIA Technical Blog: Reducing Cold Start Latency for LLM Inference with Run:ai Model Streamer
• AWS Machine Learning Blog: Accelerate LLM Model Loading with GPUDirect on Amazon FSx for Lustre