Continuous Batching in Production LLM Serving: Iteration-Level Scheduling, Chunked Prefills, and Throughput Trade-Offs

Static batching served as the standard execution paradigm for deep learning inference across computer vision and traditional natural language processing for years. In those domains, incoming requests typically feature fixed input dimensions and deterministic execution graphs. Autoregressive large language model serving breaks every assumption underlying static batching. Input prompts vary widely in token length, output generations terminate nondeterministically upon emitting an end-of-sequence token, and the execution profile bifurcates into two fundamentally distinct phases: compute-bound prompt evaluation (prefill) and memory-bandwidth-bound token generation (decode).

When static batching is applied to autoregressive generation, a batch of requests must be padded to the maximum sequence length in both prompt and completion phases. The entire batch executes synchronously, meaning fast requests that finish in ten tokens remain locked in GPU memory, idling while waiting for the slowest request in the batch to complete its multi-hundred-token generation. This structural mismatch produces severe pipeline bubbles, bloated memory allocations, and depressed hardware utilization.

Continuous batching (also referred to as iteration-level scheduling or in-flight batching) resolves this limitation by shifting the scheduling boundary from the request level to the iteration level. Rather than treating a batch as an immutable set of sequences that enter and leave execution together, a continuous batching scheduler evaluates the state of every active request at every forward pass of the model.

The Orca Architecture and Iteration-Level Scheduling

The structural foundation of continuous batching was established by Yu et al. in Orca (OSDI 2022). Orca introduced two core mechanisms: iteration-level scheduling and selective batching.

In traditional request-level batching, the inference engine executes a static loop over N sequences until all N reach termination. In contrast, iteration-level scheduling invokes the model execution engine for exactly one forward pass across the current batch of tokens. Immediately after each iteration completes:

1. The scheduler inspects the emitted tokens across all active slots.
2. Any sequence that emits a stop sequence or reaches its maximum token budget is retired from the batch, freeing its allocated key-value (KV) cache slots immediately.
3. Newly arrived requests in the waiting queue are admitted into the batch to occupy the vacated slots before the next forward pass begins.

Because attention computations require access to historical KV states for each sequence, different sequences in a continuous batch reside at different sequence positions. Orca introduced selective batching to handle this heterogeneity. While non-attention operations (such as linear projections, layer normalization, and feed-forward networks) can be executed across all tokens in the batch using unified matrix multiplications, the multi-head attention operation is dispatched using sequence-specific offsets into the KV cache.

In benchmarks presented in the original paper, Orca demonstrated up to 36.9x higher throughput compared to NVIDIA FasterTransformer running static batching at equivalent latency targets on GPT-3 class models.

Prefill-Decode Interference and Chunked Prefills

While iteration-level scheduling eliminates the idling bubbles of static batching, it exposes a secondary operational bottleneck: prefill-decode interference.

LLM inference operates in two distinct operational regimes:

• Prefill Phase (Prompt Evaluation): The model processes all prompt tokens in parallel. This phase is heavily compute-bound, saturating GPU tensor cores via high-arithmetic-intensity General Matrix Multiplies (GEMM).
• Decode Phase (Token Generation): The model generates one token per sequence autoregressively. This phase is memory-bandwidth-bound, executing General Matrix-Vector Multiplies (GEMV) where model weights and KV caches must be loaded from high-bandwidth memory (HBM) to compute just a single token per sequence.

When a standard continuous batching scheduler admits a new request with a 2,048-token prompt into a running batch of decode requests, the model must execute a large prefill GEMM in the same iteration or prior to the decode step. This compute surge introduces a substantial latency spike for the existing decode requests. For real-time applications such as interactive chat or code completion, this interference causes severe degradation in Inter-Token Latency (ITL) and jitter in Time to First Token (TTFT).

To mitigate this interference, Agrawal et al. introduced SARATHI, proposing chunked prefills and decode-maximal batching:

• Prompt Chunking: Instead of evaluating an entire prompt of L tokens in a single monolithic forward pass, the scheduler partitions the prefill into discrete chunks (for example, 512 tokens).
• Piggybacking Decodes: In each scheduling iteration, the engine co-schedules one prefill chunk alongside multiple active decode tokens. The compute-intensive prefill chunk raises the arithmetic intensity of the forward pass to saturate GPU compute units, while the decode requests piggyback on the execution without incurring dedicated memory-read overhead.
• Normalized Iteration Time: Because prefill chunks are capped at a predictable token budget, iteration execution time remains uniform, stabilizing Inter-Token Latency and eliminating extreme P99 tail spikes.

Memory Allocation and Preemption Mechanics

Continuous batching depends directly on dynamic memory management. Because the total number of tokens generated per request is unknown ahead of time, serving engines cannot pre-allocate contiguous memory buffers for maximum sequence lengths without causing severe internal fragmentation.

Modern serving engines address this challenge by integrating continuous batching with paged memory architectures, as demonstrated by vLLM and PagedAttention (Kwon et al., SOSP 2023):

• Dynamic Page Allocation: KV caches are divided into fixed-size physical memory blocks (typically 16 or 32 tokens). As each active request generates tokens across scheduling iterations, new memory blocks are allocated on demand from a global pool.
• KV Cache Exhaustion: Under heavy request concurrency, the total demand for physical KV cache blocks can exceed available GPU memory capacity.
• Preemption Strategies: When no free blocks remain to advance active sequences in an iteration, the scheduler must preempt one or more low-priority requests. Two main recovery mechanisms exist:
• Swapping: The engine transfers the preempted sequence's KV cache blocks from GPU HBM to host CPU RAM via PCIe, pausing its execution until GPU memory clears.
• Recomputation: The engine frees the preempted sequence's GPU blocks entirely and drops its state. When scheduling capacity reopens, the engine re-evaluates the original prompt and previously generated tokens in a single prefill pass to restore the KV cache. Recomputation often achieves lower overall latency overhead than host memory swapping when host-to-device PCIe bandwidth is constrained.

Production Metric Trade-Offs

Configuring a continuous batching pipeline in production requires balancing three primary performance metrics:

• Time to First Token (TTFT): The duration from request receipt to the emission of the first generated token. Prioritizing prefill requests reduces TTFT but stalls ongoing decode streams.
• Inter-Token Latency (ITL) / Time per Output Token (TPOT): The duration between successive token emissions for an active stream. High ITL variation produces noticeable stutter in streaming user interfaces.
• Aggregate Throughput (Tokens per Second): The total number of prompt and generation tokens processed per second per GPU. Maximizing throughput requires large batch sizes that saturate memory bandwidth and compute units, pushing operational points closer to GPU memory saturation.

Operational levers shape these metrics in distinct ways:

• Max Num Batched Tokens: Higher token budgets improve batch efficiency and overall throughput. Large budgets allow larger prefill chunks, lowering TTFT for long prompts, but uncapped budgets increase per-iteration variance and destabilize ITL.
• Chunked Prefill Activation: Chunking improves hardware utilization by mixing compute-bound and memory-bound operations. While it slightly increases TTFT for very long prompts due to multi-step chunking, it dramatically reduces P99 ITL spikes and stabilizes token streaming.
• Preemption via Recompute: Maximizes GPU cache utilization for active batches without CPU swap stalls. It introduces prefill recomputation overhead when traffic spikes exceed capacity, but preserves steady ITL for non-preempted streams during high load.

Engine Implementations and Deployment Landscape

Iteration-level scheduling is standard across all production-grade open-source and proprietary LLM inference runtimes:

• vLLM: Employs an asynchronous scheduling loop (vllm.core.scheduler) implementing iteration-level batching with PagedAttention and native chunked prefill.
• NVIDIA TensorRT-LLM: Implements continuous batching under the name In-Flight Batching via its C++ execution runtime, providing custom fused multi-head attention (FMHA) kernels and KV cache paged allocators optimized for NVIDIA Hopper and Blackwell architectures.
• Hugging Face Text Generation Inference (TGI): Uses a Rust-based web server and scheduler communicating with Python model execution workers over gRPC, driving continuous batching with dynamic token budget constraints.
• LMDeploy (TurboMind): Implements Persistent Batching, fusing linear layers and attention kernels with custom memory management for high-throughput deployment.

For engineering teams operating large-scale LLM infrastructure, continuous batching and chunked prefill form the baseline layer of inference efficiency, providing the scheduling foundation required before layering advanced optimizations such as speculative decoding, prefix caching, or disaggregated serving.

Sources

• Orca: A Distributed Serving System for Transformer-Based Generative Models (USENIX OSDI 2022)
• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills (arXiv:2308.16369)
• Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.05587)
• NVIDIA TensorRT-LLM In-Flight Batching Architecture
• Hugging Face Text Generation Inference Documentation