Disaggregated Prefill and Decode in Production LLM Serving: Architecture, Network KV Cache Migration, Chunked Prefill Trade-Offs, and Asymmetric Hardware Economics
Large language model serving systems have historically treated transformer execution as a homogeneous sequence of forward passes over a single unified GPU pool. Under continuous batching engines, incoming requests execute their prompt evaluation (prefill) and autoregressive token generation (decode) on the exact same accelerators, co-locating both phases within shared iteration batches. While continuous batching improves GPU compute utilization compared to static batching, co-locating prefill a
1 min