Serverless GPU Inference in Production: Cold Starts, GPU Memory Snapshotting, and Weight Paging Architectures

Deploying large language models on dedicated cloud GPUs creates an uncomfortable financial trade-off: keeping enterprise accelerators such as NVIDIA H100s or A100s warm 24/7 costs thousands of dollars per instance each month, yet scaling instances to zero introduces severe latency penalties. When traffic arrives at a dormant node, a standard inference server cold start can take anywhere from 60 seconds to over 7 minutes to return the first token.

To make scale-to-zero viable for latency-sensitive production workloads, infrastructure engineers are redesigning how model weights, CUDA runtime contexts, and kernel execution graphs are stored and initialized. Techniques such as GPU memory snapshotting, engine sleep/wake lifecycles, and direct-to-GPU weight streaming have compressed cold-start latencies by 70% to 90%, shifting serverless GPU inference from batch-only workloads into interactive serving environments.

The Four Phases of a GPU Cold Start

A cold start in GPU serving is not a single operation. It consists of four discrete stages, each bounded by different hardware bottlenecks:

+-----------------------------------------------------------------------------------+
| 1. Container Provisioning    --> 2. CUDA Context Init  --> 3. Weight Ingestion    |
| (Rootfs fetch & overlay)         (Driver handshake)        (Disk -> Host -> VRAM) |
+-----------------------------------------------------------------------------------+
                                                                     |
                                                                     v
+-----------------------------------------------------------------------------------+
| 4. Engine Warmup & Execution Prep                                                 |
| (Triton JIT compilation, CUDA graph capture across batch buckets, KV pool alloc)  |
+-----------------------------------------------------------------------------------+

1. Container Provisioning (10s to 60s)

The orchestrator must pull the container image, extract image layers, and mount the root filesystem. For standard LLM serving images containing PyTorch, CUDA libraries, and inference engines like vLLM, image sizes frequently exceed 15GB to 25GB. Without aggressive image caching on the host node, layer extraction alone accounts for multiple minutes of startup time.

2. CUDA Runtime Context Initialization (2s to 10s)

When the Python runtime initializes CUDA, the user-space driver negotiates device handles, maps virtual memory spaces, and sets up GPU context state. On multi-GPU configurations utilizing NVLink or PCIe switches, peer-to-peer memory mappings and collective communication libraries (such as NCCL) introduce additional initialization overhead.

3. Model Weight Ingestion (15s to 90s)

Model weights must move from remote storage (such as AWS S3 or distributed network volumes) to host system RAM, and then over PCIe or NVLink into GPU VRAM. For a 70-billion-parameter FP16 model (approximately 140GB of tensor data), transferring weights at typical NVMe sequential read speeds of 3 GB/s to 6 GB/s takes 25 to 45 seconds under optimal conditions. Over congested network filesystems, this stage often stretches past 90 seconds.

4. Engine Compilation, CUDA Graph Capture, and KV Allocation (15s to 60s)

Modern inference engines rely on just-in-time (JIT) compilation and pre-recorded execution graphs to maximize throughput during generation:

• Triton / TorchInductor Kernels: Engines compile fused attention and matrix multiplication kernels on the fly.
• CUDA Graph Capture: Engines capture static execution graphs across multiple batch-size buckets (e.g., batch sizes 1, 2, 4, 8, 16, 32) to eliminate CPU launch overhead during autoregressive decoding.
• KV Cache Memory Allocation: The engine profiles remaining VRAM after weight loading and claims up to 90% of available memory for the PagedAttention memory pool.

In an unoptimized configuration, these warmup steps consume significantly more time than loading the raw model weights.

GPU Memory Snapshotting: Bypassing the Warmup Pipeline

Rather than repeating driver handshakes, weight transfers, and CUDA graph captures on every cold start, infrastructure platforms are deploying GPU memory snapshotting.

Derived from Linux process checkpointing frameworks such as Checkpoint/Restore in Userspace (CRIU) and enhanced by NVIDIA driver 550+ kernel extensions, memory snapshotting captures the entire state of both CPU and GPU memory once the server is warm.

Normal Cold Start:
[ Container Init ] -> [ CUDA Init ] -> [ Load Weights ] -> [ JIT / CUDA Graphs ] -> Ready (60-460s)

Snapshot Restore:
[ Container Restore ] -> [ Direct VRAM/RAM Memory Copy ] -------------------------> Ready (2-15s)

When a snapshot is created:

1. The model server initializes completely, compiling all Triton kernels and capturing CUDA graphs.
2. The orchestrator dumps the allocated host memory pages and GPU VRAM buffers directly to local high-speed NVMe or fast object storage as a contiguous binary snapshot.
3. When new traffic arrives at an idle cluster, the serverless runtime skips container boot, Python package import, weight parsing, and CUDA graph recording. Instead, it streams the snapshot directly into memory via high-throughput memory-mapping calls.

Production benchmarks show substantial reductions in latency. Modal's memory snapshot benchmarks demonstrated a reduction in median cold start time for 3B parameter models from ~118 seconds down to ~12 seconds. On larger serving configurations, independent engineering evaluations cut cold starts from 460 seconds down to ~70 seconds by pairing GPU snapshots with engine sleep optimizations. Similarly, Cerebrium reported an average 71% cold-start reduction across workloads, with up to 88% latency drops on vLLM deployments.

Engine Dormancy and vLLM Sleep Mode

A critical challenge with memory snapshotting is managing volatile runtime state. In production serving, the Key-Value (KV) cache grows and shifts dynamically with request volume. Checkpointing an engine while active KV cache blocks occupy VRAM inflates snapshot sizes by tens of gigabytes, slowing down disk writes and snapshot restore transfers.

To solve this, modern inference engines implement explicit sleep and wake lifecycles. vLLM Sleep Mode allows the inference engine to release non-essential GPU memory allocations while keeping the underlying process and static runtime structures intact.

# Conceptual Sleep/Wake Lifecycle for Serverless Snapshots
from vllm import LLM

# 1. Initialize engine and perform warmup
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", enforce_eager=False)

# Warmup run to compile Triton kernels and capture CUDA graphs
llm.generate(["Warmup prompt to populate execution graphs"])

# 2. Enter sleep mode before taking the snapshot
# Releases KV cache memory pool while keeping compiled CUDA graphs intact
llm.sleep(level=1)

# Platform orchestrator takes memory snapshot here:
# snapshot_manager.capture_gpu_state()

# 3. On container restore, wake up the engine
llm.wake_up()

By putting the engine into sleep mode prior to snapshot capture:

• Transient KV cache tensors are discarded, reducing the snapshot to only model parameters and static CUDA context states.
• Re-compilation of kernels is avoided upon wake-up because the memory references to compiled graph binaries remain valid.
• vLLM performance data indicates that waking a dormant model is 18x to 20x faster than performing a full fresh load from scratch.

Direct-to-GPU Weight Streaming and Progressive Loading

When full memory snapshots are impractical due to frequent model updates or dynamic adapter swapping, progressive weight streaming offers an alternative path to reducing perceived cold-start latency.

Traditional loaders read entire weight files from disk into host RAM, parse tensor headers, and copy tensors to GPU VRAM via synchronous cudaMemcpy operations. This requires 2x the model footprint in system memory during loading.

Traditional Ingestion (Double Copy):
[ Storage (NVMe/S3) ] ===> [ Host System RAM ] ===> [ GPU VRAM ]
                               (Parse & Buffer)         (cudaMemcpy)

Direct-to-GPU Streaming (Zero-Copy):
[ Storage (NVMe/GPUDirect) ] ======================> [ GPU VRAM ]
                               (safetensors mmap)

Using zero-copy serialization formats like Hugging Face safetensors combined with memory-mapped files (mmap), runtimes bypass intermediate host memory buffering:

1. Header-Only Inspection: The engine reads only tensor metadata and byte offsets from the file header.
2. Direct Asynchronous Transfers: Tensors are mapped directly to virtual address ranges and transferred straight into GPU memory using asynchronous DMA (Direct Memory Access) channels.
3. Layer-Wise Execution: Under progressive weight streaming, the model server can begin executing prefill computations on initial transformer layers while trailing layers are still streaming over PCIe, reducing initial Time-To-First-Token (TTFT) for incoming requests.

Architectural Comparison: Serverless GPU Platforms

Different serverless GPU platforms implement varying architectural boundaries to balance cold start speed, security isolation, and developer workflow:

| Platform / Approach | Isolation Model | Storage & Cache Strategy | Cold-Start Optimization Mechanism | Typical Cold-Start Window (8B-70B) | | :--- | :--- | :--- | :--- | :--- | | Modal | gVisor sandboxing | Distributed network Volumes with local SSD caching | CPU & GPU Memory Snapshots + vLLM Sleep Mode | 2s to 15s (Small/Medium)<br>30s to 70s (70B) | | Baseten | Containerized Truss runtimes | Host NVMe model weight caching | Pre-warmed instance pools + persistent disk caches | 5s to 25s (Small/Medium)<br>40s to 90s (70B) | | Cerebrium | MicroVM containers | Object storage & local NVMe | Hardware-level memory snapshots (CRIU / CUDA driver 550+) | 2s to 12s (Small/Medium)<br>25s to 60s (70B) | | RunPod Serverless | Docker containers | Persistent Network Volumes | FlashBoot container layer optimization + warm workers | 10s to 35s (Small/Medium)<br>60s to 120s (70B) |

Production Failure Modes and Invalidation Traps

Deploying memory snapshots and weight streaming in production introduces specific architectural pitfalls that teams must safeguard against:

1. Hardware Topology Locks

CUDA execution graphs and compiled binary kernels are strictly coupled to specific GPU microarchitectures and SM (Streaming Multiprocessor) counts. A snapshot generated on an NVIDIA Hopper H100 (SM 9.0) will fail to restore on an NVIDIA Ampere A100 (SM 8.0) or Ada Lovelace L40S (SM 8.9). Serverless schedulers must enforce strict hardware-affinity tags, ensuring snapshot restore requests are routed exclusively to matching GPU compute classes.

2. Network Volume Cache Desynchronization

When an engine running inside a snapshot holds open file descriptors to cache directories on attached network volumes (such as torch.compile cache files), any mismatch between the snapshot state and the mounted filesystem state triggers abrupt runtime failures. If the remote volume was unmounted or modified while the instance was sleeping, filesystem path verification fails upon restore, forcing the engine into a slow fallback crash-and-reboot cycle.

3. Sockets and Local Coordination State

In multi-GPU setups running tensor parallelism across processes, communication handles, Unix domain sockets, and shared memory segments can break across snapshot boundaries. If inter-process coordination handles become stale during a restore, the main process may report ready while worker ranks remain unresponsive. Robust orchestrators isolate inter-process communication state into re-initializable hooks triggered immediately after memory unfreezing.

4. Concurrency Thresholding and Scale-Up Flapping

In scale-to-zero environments, sudden bursts of traffic can trigger concurrent cold starts across multiple nodes before the first node has completed its snapshot restore. Infrastructure configurations must calibrate target_concurrency metrics and request queue backpressure. Routing initial traffic spikes through centralized queue buffers prevents cascading duplicate instance spawns.

Sources

• Modal Documentation: Memory Snapshots and vLLM Sleep Mode
• Modal Blog: 10x Faster Cold Starts with GPU Snapshotting
• Cerebrium Engineering: Reducing GPU Cold Starts with Memory Snapshots
• vLLM Documentation: Sleep Mode Architecture and Lifecycle
• vLLM Blog: Zero-Reload Model Switching with vLLM Sleep Mode
• GitHub: vLLM CUDA Checkpoint / Restore RFC #34303
• Spheron Engineering: GPU Cold Start on Serverless LLM Inference