Disaggregated Prefill and Decode (PD Disaggregation) in Production LLM Serving: Comparing DistServe, Mooncake, Splitwise, and vLLM Architecture, KV Cache RDMA Transfer, TTFT/ITL Decoupling, and Cluster Economics

In conventional large language model (LLM) inference engines, prompt processing (prefill) and autoregressive token generation (decode) execute on the same physical GPU workers. While colocated serving simplifies cluster orchestration, it creates a fundamental architectural contradiction: prefill is compute-bound, saturating tensor cores with high arithmetic intensity, while decode is memory-bandwidth-bound, stalled on high-bandwidth memory (HBM) lookups at low arithmetic intensity.

When colocated in production, long-context prefill batches preempt active decoding iterations. This phase interference induces head-of-line blocking, causing severe tail-latency spikes in Inter-Token Latency (ITL) and Time to First Token (TTFT).

To resolve this bottleneck, production serving infrastructure is shifting toward Prefill-Decode (PD) Disaggregation. By decoupling prefill workers from decode workers and streaming the generated Key-Value (KV) cache across high-speed interconnects, engineering teams can optimize hardware allocation, parallelism strategies, and Service Level Objectives (SLOs) independently for each phase.

1. The Mechanics of Phase Interference in Colocated Serving

Understanding why colocated serving degrades under load requires examining the mathematical and operational differences between the two execution phases of the transformer forward pass:

1. Prefill Phase (Prompt Ingestion):

• Characteristics: Parallel processing of all $S_{\text{prompt}}$ input tokens simultaneously.
• Compute Profile: Compute-bound Matrix-Matrix multiplications (GEMM). Arithmetic intensity scales linearly with prompt length ($\mathcal{O}(S_{\text{prompt}})$), reaching high utilization on modern tensor cores.
• Primary Metric: Time to First Token (TTFT).

1. Decode Phase (Autoregressive Token Generation):

• Characteristics: Sequential token generation where step $t$ depends on token $t-1$.
• Compute Profile: Memory-bandwidth-bound Matrix-Vector multiplications (GEMV). Arithmetic intensity is minimal (typically 1 to 2 FLOPs per byte transferred from HBM).
• Primary Metric: Time Per Output Token (TPOT) or Inter-Token Latency (ITL).

The Colocation Dilemma

In continuous batching schedulers (such as baseline Orca or vLLM deployments), incoming prompts are batched together with ongoing decode requests. When a request with an 8,000-token prompt enters the batch, the prefill computation consumes the GPU tensor cores for hundreds of milliseconds.

During this prefill window, all concurrent decode streams assigned to that worker are paused. The result is a sharp bimodal distribution in ITL: while uncontended decode steps take 15 to 25 milliseconds, steps interrupted by prefill burst to 200 to 500 milliseconds.

Techniques like chunked prefill (SARATHI-Serve) split long prompts into smaller token chunks (e.g., 512 tokens) to interleave with decode iterations. While chunked prefill reduces extreme ITL outliers, it introduces compute inefficiency by repeatedly reloading model weights for small prefill chunks, fails to eliminate cross-phase GPU memory fragmentation, and forces both phases to share identical tensor parallelism configurations.

2. Phase Splitting Fundamentals

Phase splitting, pioneered by Splitwise (Patel et al., ISCA 2024) and formalized by DistServe (Zhong et al., OSDI 2024), physically isolates the prefill and decode stages into distinct worker pools connected via high-bandwidth interconnects (PCIe Gen5, RoCE v2, or InfiniBand).

Incoming Request
       │
       ▼
┌──────────────┐      High-Bandwidth RDMA / PCIe      ┌──────────────┐
│ Prefill Pool │ ───────────────────────────────────► │ Decode Pool  │
│  (Compute)   │        KV Cache Data Stream          │   (Memory)   │
└──────────────┘                                      └──────────────┘
  • High Tensor Parallelism (TP)                        • Wide Pipeline / Data Parallelism
  • FP8/FP16 Compute Density (H100/B200)                • High HBM Capacity & Bandwidth
  • Optimized for TTFT SLO                              • Optimized for ITL / TPOT SLO

Divergent Parallelism Strategies

Disaggregation unlocks independent parallelism schemes tailored to phase characteristics:

• Prefill Workers: Benefit from aggressive Tensor Parallelism (e.g., $\text{TP}=4$ or $\text{TP}=8$). Because prefill is compute-bound, sharding the attention and MLP matrix multiplications across GPUs directly divides TTFT latency by the TP degree.
• Decode Workers: Benefit from Pipeline Parallelism (PP) or wide Data Parallelism (DP) with minimal Tensor Parallelism (e.g., $\text{TP}=1$ or $\text{TP}=2$). Because decode is memory-bound, cross-GPU all-reduce communication overhead in high TP configurations can exceed matrix computation time for batch size 1. Smaller TP degrees with larger batch sizes maximize memory bandwidth saturation and token throughput.

3. Comparison of Leading Disaggregated Frameworks

Four primary architectures define modern disaggregated LLM serving:

| Architecture | Origin & Venue | Core Paradigm | KV Cache Transfer Engine | Parallelism Strategy | Key Strength / Trade-off | | :--- | :--- | :--- | :--- | :--- | :--- | | DistServe | Peking Univ. / OSDI 2024 | Dual-SLO Attainment Optimization | Point-to-point TCP/RDMA streaming | Independent TP/PP per phase via offline profiling | Maximizes goodput under strict TTFT and TPOT latency targets | | Mooncake | Kuaishou / Tsinghua / FAST 2025 | KVCache-Centric Memory Architecture | NIXL over RDMA with hierarchical storage pool | Multi-instance $X_p \times Y_d$ mesh routing | Unified tiered KV pool (HBM, DRAM, SSD) with global chunk reuse | | Splitwise | Microsoft / ISCA 2024 | Heterogeneous Hardware Phase Splitting | Network-streamed KV transfers | Phase-split node allocation with dynamic migration | Evaluates heterogeneous compute (dense compute nodes vs memory nodes) | | vLLM P/D Disaggregation | vLLM Community / vLLM Docs | Layer-by-Layer Pipelined Engine | Pluggable Connectors (NIXLConnector, LMCache) | Decoupled Ray / distributed worker instances | Layer-by-layer overlapping hides transfer latency behind compute |

4. Deep-Dive into Production Implementations

DistServe: SLO-Driven Resource Co-Optimization

DistServe treats LLM serving as a constrained optimization problem: maximize per-GPU "goodput" (the maximum request rate served while satisfying both TTFT and TPOT SLO thresholds at the 99th percentile).

DistServe incorporates a cluster profiling simulator that evaluates application traffic patterns (prompt-to-output ratios, arrival distributions) and determines:

1. The exact ratio of prefill GPUs ($N_p$) to decode GPUs ($N_d$).
2. The optimal parallelism configuration for each phase ($\text{TP}_p, \text{PP}_p$ vs $\text{TP}_d, \text{PP}_d$).
3. Placement strategies that co-locate prefill and decode workers sharing high inter-node network bandwidth to minimize KV transfer latency.

Empirical evaluations across Llama-2-70B and OPT-66B benchmarks showed DistServe achieving up to 4.48x higher request throughput and up to 10.2x higher SLO compliance compared to colocated vLLM baselines.

Mooncake: A KVCache-Centric Disaggregated Architecture

Developed by Kuaishou and Tsinghua, Mooncake represents a paradigm shift: instead of treating KV cache transfer merely as a communication step between compute instances, Mooncake builds a Disaggregated KVCache Pool.

┌─────────────────────────────────────────────────────────────┐
│                    Mooncake Conductor                       │
│           (Global Request & KV-Chunk Scheduler)            │
└──────────────────────────────┬──────────────────────────────┘
                               │
               ┌───────────────┴───────────────┐
               ▼                               ▼
       ┌───────────────┐               ┌───────────────┐
       │ Prefill Nodes │               │ Decode Nodes  │
       │  (Instances)  │               │  (Instances)  │
       └───┬───────▲───┘               └───┬───────▲───┘
           │       │                       │       │
           │       │      NIXL (RDMA)      │       │
           ▼       └───────────────────────▼       │
   ┌───────────────────────────────────────────────┴──┐
   │            Distributed KVCache Pool              │
   │  ┌───────────────┐ ┌───────────────┐ ┌─────────┐  │
   │  │ GPU HBM Cache │ │ CPU DRAM Pool │ │ SSD/NVMe│  │
   │  └───────────────┘ └───────────────┘ └─────────┘  │
   └──────────────────────────────────────────────────┘

Key innovations of Mooncake include:

• Hierarchical Storage Tiering: Aggregates underutilized CPU DRAM and local NVMe SSDs across the inference cluster into a unified distributed KV cache store, extending beyond GPU HBM.
• NIXL (Network Inference Xfer Library): An asynchronous communication library built on UCX and kernel bypass RDMA, capable of saturating 400Gbps/800Gbps network interfaces without CPU intervention.
• Dynamic Chunk Prefetching: When a multi-turn agentic query shares a system prompt or multi-turn history, Mooncake locates pre-existing KV chunks in the distributed pool and streams them directly into the assigned decode node, skipping the prefill phase entirely.
• In production deployment at Kuaishou serving hundreds of millions of daily requests, Mooncake reduced P99 latency by over 40% while handling 75% more concurrent requests under identical GPU budgets.

vLLM P/D Disaggregation and Layer-by-Layer Pipelining

In vLLM, disaggregated prefilling is implemented within vllm/distributed/kv_transfer via modular connectors such as NIXLConnector and LMCacheConnector.

Rather than executing the full prefill pass across all $L$ layers and then transmitting the complete multi-gigabyte KV cache tensor in a single bulk transfer, modern vLLM pipelines utilize layer-by-layer pipelined transfer:

Prefill Worker:   [ Layer 1 Compute ] ──► [ Layer 2 Compute ] ──► [ Layer 3 Compute ]
                         │                       │                       │
                         ▼ (Async RDMA Send)     ▼ (Async RDMA Send)     ▼ (Async RDMA Send)
Network:          [ Layer 1 KV Xfer ] ──► [ Layer 2 KV Xfer ] ──► [ Layer 3 KV Xfer ]
                         │                       │                       │
                         ▼ (Async RDMA Recv)     ▼ (Async RDMA Recv)     ▼ (Async RDMA Recv)
Decode Worker:    [ Layer 1 Ready   ] ──► [ Layer 2 Ready   ] ──► [ Ready for Token 1 ]

As soon as layer $l$ completes its attention and feed-forward pass on the prefill instance, its KV cache is immediately scheduled for non-blocking asynchronous transmission over RDMA. While the network transfers layer $l$, the prefill GPU continues computing layer $l+1$. By the time the final layer completes prefill, earlier layers are already loaded into the decode worker's HBM, effectively masking network transfer latency.

5. KV Cache Transfer Mechanics: Sizing and Latency Budgets

The technical viability of PD disaggregation depends strictly on whether KV cache transfer time over the network remains smaller than prefill computation time.

Sizing the KV Cache Payload

For a transformer model operating with Grouped-Query Attention (GQA), the byte size $M_{\text{kv}}$ of the KV cache generated for a prompt of sequence length $S$ is calculated as:

$$M_{\text{kv}} = 2 \times L \times H_{\text{kv}} \times D_{\text{head}} \times S \times P$$

Where:

• $2$: Accounts for both Key and Value tensors.
• $L$: Total number of transformer layers.
• $H_{\text{kv}}$: Number of KV attention heads.
• $D_{\text{head}}$: Dimension per attention head ($D_{\text{model}} / H_{\text{attn}}$).
• $S$: Sequence length in tokens.
• $P$: Precision in bytes per parameter (2 bytes for FP16/BF16, 1 byte for FP8).

Practical Sizing Example: Llama-3-70B

For Llama-3-70B ($L = 80$, $H_{\text{kv}} = 8$, $D_{\text{head}} = 128$, FP16 precision $P = 2$):

$$M_{\text{kv}} = 2 \times 80 \times 8 \times 128 \times S \times 2 = 327,680 \times S \text{ bytes} \approx 320 \text{ KB per token}$$

For an 8,192-token prompt:

• Total KV Cache Size: $8,192 \times 320\text{ KB} = 2.56\text{ GB}$.

Network Transfer Latency vs. Prefill Compute Time

Let us evaluate the transfer time of a 2.56 GB KV cache across standard interconnects:

| Interconnect Standard | Unidirectional Bandwidth | Raw Transfer Latency (2.56 GB) | Prefill Time (H100, 8k Tokens) | Transfer Overlap Ratio | | :--- | :--- | :--- | :--- | :--- | | PCIe Gen5 x16 (Intra-Node) | 64 GB/s | ~40.0 ms | ~95.0 ms | Fully Masked (100%) | | 400 Gbps RoCE v2 / InfiniBand | 50 GB/s (effective ~45 GB/s) | ~56.8 ms | ~95.0 ms | Fully Masked (100%) | | 800 Gbps InfiniBand / ConnectX-7 | 100 GB/s (effective ~90 GB/s) | ~28.4 ms | ~95.0 ms | Fully Masked (100%) | | 100 Gbps Ethernet (Legacy) | 12.5 GB/s (effective ~11 GB/s) | ~232.7 ms | ~95.0 ms | Network Bound (2.4x slowdown) |

On 400Gbps and 800Gbps fabrics, layer-by-layer pipelining ensures that network transfer completes concurrently with prefill execution, introducing near-zero incremental latency to TTFT.

Furthermore, quantizing the KV cache to FP8 (via native kernel support in vLLM and TensorRT-LLM) reduces the transfer volume from 2.56 GB to 1.28 GB, cutting network transmission time in half with negligible impact on generation perplexity.

6. Cluster Economics and Deployment Guidelines

PD Disaggregation is not a universal replacement for colocated serving; its economic advantage depends directly on the traffic distribution and context characteristics of the application workload.

                     ┌───────────────────────────────┐
                     │ Workload Assessment Framework │
                     └───────────────┬───────────────┘
                                     │
           ┌─────────────────────────┴─────────────────────────┐
           ▼                                                   ▼
┌─────────────────────────────────────┐     ┌─────────────────────────────────────┐
│  Prompt Length / Output Ratio > 4:1 │     │ Prompt Length / Output Ratio < 1:1  │
│    (RAG, Repo Analysis, Extraction) │     │ (Creative Chat, Synthetic Reasoning)│
└──────────────────┬──────────────────┘     └──────────────────┬──────────────────┘
                   │                                           │
                   ▼                                           ▼
┌─────────────────────────────────────┐     ┌─────────────────────────────────────┐
│   Deploy Disaggregated Architecture │     │  Deploy Colocated Serving + Chunking│
│  • Dedicate 2:1 or 3:1 Prefill:Dec  │     │  • Continuous batching with 512-tok  │
│  • Enable Layer-by-Layer RDMA xfer  │     │    chunked prefill is cost-optimal   │
│  • 3x-4x Goodput gain at tight SLO  │     │  • Avoids inter-node network costs   │
└─────────────────────────────────────┘     └─────────────────────────────────────┘

When to Deploy Disaggregated Serving

1. Strict Tail-Latency (P99) SLO Requirements: Applications requiring bounded ITL (< 25 ms per token) for interactive real-time voice or enterprise coding assistants where prefill spikes cannot be tolerated.
2. Asymmetric Long-Prompt Workloads: Retrieval-Augmented Generation (RAG), long-document analysis, repository-scale code generation, and multi-turn agent loops where prompt tokens exceed generation tokens by 4:1 or more.
3. High Cluster Concurrency: Deployments operating dozens of GPUs where dedicating 4 to 8 nodes to a specialized prefill pool and 12 to 16 nodes to a decode pool eliminates cross-phase resource starvation.
4. InfiniBand / RoCE Fabric Availability: Infrastructure equipped with 400Gbps or faster inter-node interconnects capable of zero-copy RDMA transfers.

When to Retain Colocated Serving

1. Short Prompt, Long Generation Workloads: Synthetic reasoning chains, math problem generation, or code generation where input prompts are small (100 to 300 tokens) and outputs exceed 2,000 tokens. In these workloads, prefill accounts for under 5% of total compute time.
2. Bandwidth-Constrained Networks: Environments operating on standard 10Gbps or 25Gbps Ethernet networks where KV cache transfer latency exceeds prefill computation time, turning disaggregation into an operational bottleneck.
3. Small-Scale Deployments (1 to 4 GPUs): Clusters where resource partitioning creates idle capacity during uneven arrival bursts.

Sources

• DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving (OSDI 2024)
• Mooncake: A KVCache-Centric Disaggregated Architecture for LLM Serving (FAST 2025)
• Splitwise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024)
• vLLM Documentation: Disaggregated Prefilling Architecture
• LMCache: Distributed KV Cache Storage Architecture
• SARATHI-Serve: Chunked-Prefills and Piggyback Decoding for LLM Serving (OSDI 2024)