Continuous Batching

Continuous Batching in Production LLM Serving: Iteration-Level Scheduling, Chunked Prefills, and Throughput Trade-Offs

Static batching served as the standard execution paradigm for deep learning inference across computer vision and traditional natural language processing for years. In those domains, incoming requests typically feature fixed input dimensions and deterministic execution graphs. Autoregressive large language model serving breaks every assumption underlying static batching. Input prompts vary widely in token length, output generations terminate nondeterministically upon emitting an end-of-sequence t