Serving

Disaggregated Prefill and Decode in Production: Architecture, Economics, and KV Transfer Protocols

In production large language model serving, the fundamental architectural tension lies between two computationally distinct phases: prefill (processing the input prompt) and decode (generating output tokens autoregressively). In standard co-located serving systems, both phases share the same GPU instances, memory pools, and execution batches. This co-location creates head-of-line blocking, degrades Time to First Token (TTFT), inflates Time Per Output Token (TPOT), and limits cluster-wide hardwar