GPU Cluster Scheduling in Production: Slurm vs. Kubernetes (Kueue/Volcano) vs. Ray

Modern AI infrastructure represents a radical departure from traditional cloud computing. Standard cloud workloads (such as stateless microservices, web applications, and independent batch jobs) rely on fine-grained elasticity, independent container scheduling, and horizontal autoscaling. In contrast, distributed large language model (LLM) training and high-throughput inference pipelines violate virtually every assumption built into general-purpose cloud schedulers.

A distributed 70B+ parameter pre-training or fine-tuning run spanning 512 GPUs cannot make incremental progress if 511 GPUs are active while the last GPU waits in a queue. Because gradient synchronization primitives like AllReduce and All-to-All require synchronous communication across all ranks, distributed training jobs demand strict all-or-nothing allocation, low-latency network topology packing across InfiniBand or RoCE fat-tree switches, and multi-tenant fair-share arbitration.

Three primary paradigms have emerged to govern GPU cluster orchestration: High-Performance Computing (HPC) workload managers like Slurm, cloud-native batch admission schedulers on Kubernetes like Kueue and Volcano, and application-level dynamic task execution runtimes like Ray. Understanding their architectural boundaries, trade-offs, and operational sweet spots is critical for any team architecting AI compute infrastructure.

1. Slurm: High-Throughput Bare-Metal HPC Scheduling

Originally developed for scientific supercomputers, Slurm (Simple Linux Utility for Resource Management) remains the industry benchmark for raw scheduling performance, bare-metal GPU efficiency, and deterministic execution on static compute fabrics.

Architecture and Mechanics

Slurm utilizes a centralized controller architecture:

• slurmctld: The central management daemon that monitors cluster resources, maintains job queues, computes multi-factor priorities, and allocates compute nodes.
• slurmd: A lightweight daemon running on each compute node that monitors local hardware, launches tasks, and reports health state.
• slurmdbd: An optional database daemon backed by MySQL/MariaDB for historical accounting, bank allocations, and fair-share tracking.

Because Slurm is implemented entirely in C with direct socket communication, its control plane exhibits near-zero serialization overhead. While Kubernetes control planes can struggle with etcd write contention when handling thousands of rapid object state transitions, slurmctld routinely processes thousands of job submissions per second with sub-millisecond dispatch latency across clusters exceeding 100,000 CPU and GPU cores.

Core Strengths in AI Training

1. Native Gang Scheduling and Backfill Optimization: Slurm natively enforces all-or-nothing allocation. When a multi-node job requests 64 nodes with 8 H100 GPUs each, Slurm calculates the exact reservation window. If those resources are currently occupied, Slurm calculates when the high-priority job will start and uses a backfill scheduling algorithm to slot smaller, short-duration jobs into idle capacity holes without delaying the high-priority reservation.
2. Deterministic Network Topology Mapping: Slurm reads physical network architecture through a topology.conf file. By defining leaf switches, spine switches, and core network fabrics, Slurm prioritizes node allocations within the same leaf switch (minimizing InfiniBand hop count and eliminating cross-spine bisection bandwidth bottlenecks).
3. Multi-Factor Priority and Hierarchical FairShare: Slurm calculates job priority using a composite formula combining job age, partition priority, Quality of Service (QoS) tiers, and a FairShare factor. The FairShare algorithm models organizational hierarchies, decaying past GPU usage over half-life curves to ensure teams that underutilized their quota receive instantaneous scheduling preference.
4. Container Integration via Pyxis and Enroot: NVIDIA developed Enroot and the Pyxis Slurm plugin to bridge Docker container images into bare-metal Slurm workflows without running a privileged Docker daemon on compute nodes.

Operational Limitations

• Static Infrastructure Mindset: Slurm was designed for fixed bare-metal hardware. It does not provide native primitives for dynamic cloud VM autoscaling, ephemeral disk provisioning, or spot instance lifecycle management.
• Rigid Multi-Tenancy: Slurm relies on host-level POSIX UIDs, GIDs, and LDAP/Kerberos identity management. Providing isolated network namespaces or granular role-based access control (RBAC) requires significant custom engineering.
• Inference Inflexibility: Slurm excels at static, long-running batch jobs, but it is poorly suited for dynamic microservices, real-time model inference serving, or HTTP ingress routing.

2. Kubernetes: Cloud-Native GPU Orchestration with Kueue and Volcano

Kubernetes provides standard enterprise infrastructure management: rich REST APIs, declarative desired-state reconciliation, cloud provider integration, container runtime interfaces (CRI), and automated rollouts. However, vanilla Kubernetes was architected for stateless web applications, making its default scheduler problematic for distributed AI.

The Default Scheduler Problem

The standard kube-scheduler evaluates Pods sequentially in isolation. In a distributed training job requiring 8 worker pods (each requesting 8 GPUs across separate nodes):

1. kube-scheduler schedules Pod 1 through Pod 7 onto available nodes.
2. When attempting to schedule Pod 8, no node in the cluster has 8 available GPUs.
3. Pod 8 enters a Pending state.
4. Because Pods 1 through 7 are already running and occupying 56 GPUs, they block other jobs from starting.
5. Because the training script waits indefinitely at rank synchronization for Pod 8, the entire cluster deadlocks, wasting hundreds of GPU hours.

To eliminate this deadlock and add enterprise batch management, the cloud-native ecosystem developed two primary solutions: Kueue and Volcano.

Kueue: Out-of-Band Admission Control and Job Queueing

Kueue is an official Kubernetes Special Interest Group (SIG) project that manages batch queues as an admission controller rather than replacing kube-scheduler. Kueue operates at the job abstraction level, intercepting batch objects before their underlying pods are submitted to the core scheduler.

+-------------------------------------------------------------------+
|                        Workload Submission                        |
|        (JobSet, RayJob, Kubeflow MPIJob, PyTorchJob, TrainJob)     |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                       Kueue Admission Layer                       |
|  - LocalQueue & ClusterQueue Quotas                               |
|  - Cohort Borrowing / Preemption Rules                            |
|  - Topology-Aware Scheduling (TAS) Placement Generation           |
|  - All-or-Nothing (Gang) Admission Gate Release                   |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                      Core kube-scheduler                          |
|         (Binds Pods to Nodes based on Kueue's Node Assignment)     |
+-------------------------------------------------------------------+

Key Architecture and Features of Kueue:

1. Separation of Concerns: Kueue does not assign pods to nodes directly. Instead, it manages queue priority, nominal quotas, fair sharing, and workload admission. Once Kueue admits a workload by unsuspending it, the standard kube-scheduler performs the actual pod-to-node binding.
2. Hierarchical Cohorts and Dynamic Quota Borrowing: Kueue organizes ClusterQueues into Cohorts. If Team A has a nominal quota of 128 GPUs and Team B has 128 GPUs, Team A can automatically borrow Team B's idle GPUs when Team B is inactive. When Team B submits new workloads, Kueue can reclaim or preempt the borrowed capacity according to configurable reclamation policies.
3. Topology-Aware Scheduling (TAS): Kueue TAS allows administrators to map physical datacenter network topology (racks, leaf groups, and NVLink domains) into Kubernetes custom resources. When admitting a JobSet, Kueue computes optimal node placement slices to minimize intra-job network latency.
4. Ecosystem Compatibility: Kueue natively supports standard batch custom resource definitions (CRDs), including Kubernetes Job, JobSet, LeaderWorkerSet, Kubeflow MPIJob and PyTorchJob, and KubeRay RayJob.

Volcano: Unified Batch Scheduling Engine

Unlike Kueue, Volcano is a CNCF incubating project that acts as a custom scheduler replacement for Kubernetes. Volcano incorporates batch scheduling algorithms directly into the pod-binding phase.

• Integrated Plugin Framework: Volcano executes extensible plugins during every scheduling cycle, including gang (all-or-nothing scheduling), drf (Dominant Resource Fairness across multi-dimensional CPU/GPU/memory allocations), sla (Service Level Agreement job pacing), and task-topology (affinity scoring between dependent tasks).
• Native Job Controller: Volcano provides its own vcjob (Volcano Job) primitive, which integrates multi-task lifecycles (e.g. coordinating parameter servers and workers within a single YAML manifest).

Kubernetes Batch Trade-Offs

• Control Plane Overhead: Managing tens of thousands of short-lived pods creates heavy write traffic on etcd and the API server. High-throughput batch workloads require careful API server tuning and event pruning.
• Complexity: Operating production Kubernetes GPU clusters requires a deep stack of operators: NVIDIA GPU Operator, Network Operator (for SR-IOV, RoCE, and InfiniBand), Kueue/Volcano, and storage CSI drivers.

3. Ray: Application-Level Dynamic Task and Actor Scheduling

While Slurm and Kubernetes operate at the machine and container infrastructure layers, Ray is an open-source distributed compute framework that schedules execution graphs dynamically at the Python application level.

+-------------------------------------------------------------------+
|                        Python ML Application                      |
|       (Ray Train, Ray Data, vLLM / Ray Serve, RLlib / GRPO)       |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                      Ray Core Distributed Engine                  |
|  - Dynamic Directed Acyclic Graphs (Tasks & Stateful Actors)      |
|  - Object Store (Shared-Memory Apache Arrow Plasma Store)         |
|  - Resource Placement Groups (STRICT_PACK, STRICT_SPREAD, etc.)   |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                     Underlying Infrastructure                     |
|           (Bare-Metal Slurm Cluster OR Kubernetes via KubeRay)     |
+-------------------------------------------------------------------+

Architecture and Scheduling Primitives

Ray abstracts a cluster into a unified pool of logical resources (e.g., num_cpus, num_gpus, custom resource tags):

• GCS (Global Control Service): Manages cluster-level metadata, actor registration, and node heartbeats.
• Raylet: A local scheduler daemon running on every node that manages local worker processes and coordinates zero-copy shared-memory object transfers via the Plasma store.
• Placement Groups: Ray provides explicit primitives to control where actors and tasks land across nodes:
• STRICT_PACK: All requested bundles must be placed on the exact same physical node (essential for multi-GPU single-node tensors).
• PACK: Bundles are packed onto as few nodes as possible.
• STRICT_SPREAD: Every bundle must land on a distinct physical node (critical for fault-tolerant replicas).
• SPREAD: Bundles are distributed across different nodes when capacity permits.

Why AI Teams Build on Ray

1. Heterogeneous End-to-End Pipelines: Modern post-training pipelines rarely involve pure matrix multiplication. A reinforcement learning from verifiable rewards (RLVR) or GRPO pipeline requires:

• High-throughput streaming data ingestion and tokenization (Ray Data).
• Dynamic model rollout generation and vLLM inference across multiple GPUs (Ray Serve / Actor pools).
• Distributed reward model verification and critic evaluations.
• Distributed policy gradient backpropagation (Ray Train with PyTorch FSDP).

Ray allows all four stages to execute seamlessly inside a single distributed application without dumping intermediate tensors to disk.

1. Zero-Copy Memory Sharing: When multiple actor processes on the same machine access large datasets or reference model weights, Ray's local shared-memory Plasma store allows worker processes to read shared memory structures with zero serialization overhead.

The KubeRay Bridge

Ray is not a replacement for infrastructure orchestration; it does not manage host operating systems, bare-metal networking, or physical server provisioning. In production enterprise environments, teams deploy Ray on top of Kubernetes using the KubeRay operator.

By pairing KubeRay with Kueue, infrastructure teams get the best of both worlds:

• Kueue manages organizational GPU quotas, team priorities, fair sharing, and all-or-nothing cluster admission.
• KubeRay spins up the admitted Ray head and worker pods.
• Ray Core manages fine-grained Python task scheduling, inter-worker tensor communication, and actor lifecycles within the allocated cluster.

4. Architecture and Performance Comparison Matrix

| Feature | Slurm (SchedMD) | Kubernetes + Kueue | Ray (KubeRay / Ray Core) | | :--- | :--- | :--- | :--- | | Architectural Layer | Bare-metal HPC resource manager | Cloud-native batch admission & queue manager | Distributed application & task graph runtime | | Scheduling Unit | Jobs and Job Steps (sbatch, srun) | Workloads / JobSets / Pod groups | Python Tasks and Stateful Actors | | Gang Scheduling Mechanism | Native all-or-nothing allocation with backfill | Workload admission gates (releases all pods together) | Placement Groups (STRICT_PACK / STRICT_SPREAD) | | Network Topology Awareness | Static switch tree via topology.conf | Dynamic Topology-Aware Scheduling (TAS) CRDs | Application-defined node and accelerator bundles | | Scheduling Latency | Sub-millisecond (C-native, in-memory) | 100ms - 2s (API server + etcd + admission loop) | Microseconds (Raylet local task dispatch) | | Multi-Tenancy & Quotas | Hierarchical FairShare trees, bank accounts | ClusterQueues, Cohorts, and borrowing policies | Soft limits within a cluster; relies on K8s for isolation | | Autoscaling Capability | Primarily static; custom cloud burst scripts | Native via Cluster Autoscaler and Karpenter | Dynamic worker scaling via Ray Autoscaler / KubeRay | | Ecosystem & Workloads | Pre-training, MPI, C/C++/Fortran, Pyxis PyTorch | Enterprise AI platform, JobSets, multi-model serving | Post-training, RL loops, Ray Data, vLLM serving |

5. Production Architectural Recommendations

Selecting the right scheduling tier depends on team workflows, cluster scale, and workload heterogeneity.

                                WORKLOAD PROFILE
                                       |
          +----------------------------+----------------------------+
          |                                                         |
          v                                                         v
Single monolithic pre-training                         Heterogeneous AI lifecycle
(Static GPU cluster, pure PyTorch,                     (Data prep, RL post-training,
dedicated multi-month training)                        multi-tenant teams, serving)
          |                                                         |
          v                                                         v
   [ SLURM BARE-METAL ]                                   [ KUBERNETES CONTROL PLANE ]
   - Sub-millisecond dispatch                                       |
   - topology.conf packing                                          v
   - Enroot / Pyxis containers                            [ KUEUE ADMISSION LAYER ]
                                                          - Cohort GPU borrowing
                                                          - Topology-Aware Scheduling
                                                          - Gang admission gates
                                                                    |
                                                  +-----------------+-----------------+
                                                  |                                   |
                                                  v                                   v
                                         [ JobSet / PyTorchJob ]               [ KubeRay Cluster ]
                                         Standard distributed training         Multi-stage RL / Ray Train

When to Choose Bare-Metal Slurm

• Foundation Pre-Training on Dedicated Hardware: If your organization operates a dedicated, homogenous cluster of 512+ GPUs solely running large-scale pre-training runs with standard Megatron-LM, FSDP, or DeepSpeed, Slurm provides the lowest operational overhead, the most deterministic InfiniBand topology packing, and zero control-plane jitter.

When to Choose Kubernetes with Kueue

• Multi-Tenant Enterprise Platforms: If your cluster serves multiple research, engineering, and product teams sharing a common GPU pool across training, fine-tuning, and batch inference, Kubernetes with Kueue provides robust RBAC, dynamic quota borrowing across cohorts, and native cloud autoscaling.
• Mixed Service and Batch Environments: When the same infrastructure must run long-running model endpoints (Triton, vLLM, SGLang) alongside scheduled batch jobs, Kubernetes provides a single unified control plane.

When to Deploy the Hybrid Stack (Kubernetes + Kueue + KubeRay)

• Complex Agentic and RL Training Pipelines: For modern post-training workflows involving multi-step synthetic data generation, actor-critic rollouts, reward model scoring, and parameter updates, running Ray on top of Kubernetes (orchestrated by KubeRay and admitted via Kueue) delivers maximum developer velocity without compromising enterprise quota governance.

Sources

• Slurm Workload Manager Documentation (SchedMD)
• Kueue: Kubernetes-Native Job Queueing (Kubernetes SIGs)
• Topology-Aware Scheduling with JobSet in Kueue (Kubernetes SIGs)
• Volcano: High-Performance Batch Scheduling System (CNCF)
• Ray Architecture and Placement Groups Overview (Ray Project)
• Gang Scheduling with RayJob and Kueue (Ray Documentation)
• KubeRay Operator Documentation (Ray Project)
• NVIDIA Pyxis Container Plugin for Slurm (GitHub)