Serving large language models in production has moved beyond simple iteration-level scheduling. While basic continuous batching solved the fundamental problem of GPU starvation during autoregressive decoding, modern production demands have shifted the engineering bottleneck toward KV cache memory reuse, compilation overhead, and structured generation efficiency.
Three dominant serving runtimes currently lead high-throughput deployments: vLLM, SGLang, and NVIDIA TensorRT-LLM. Each framework approaches memory virtualization, kernel execution, and request scheduling from distinct architectural assumptions.
KV Cache Architecture: PagedAttention vs. RadixAttention
The primary memory challenge during autoregressive decoding is the key-value (KV) cache. In traditional naive memory allocation, serving engines pre-allocate contiguous memory buffers sized to the model's maximum context length. This strategy causes internal and external memory fragmentation that wastes 60% to 80% of available GPU VRAM.
vLLM introduced PagedAttention, adapting the classical operating system concept of virtual memory paging to transformer KV states. PagedAttention divides the KV cache into fixed-size physical memory blocks (typically 16 or 32 tokens). Logical blocks are mapped to non-contiguous physical pages via a block table, reducing memory waste to under 4%. For prefix sharing, vLLM computes cryptographic hashes over token sequences in completed blocks, allowing distinct requests that share identical prefix blocks to point to the same physical memory addresses.
In contrast, SGLang designed RadixAttention, which manages the KV cache as an explicit radix tree (trie) data structure over token sequences. Rather than indexing memory via flat block hash tables, the radix tree natively reflects the branching topology of language model interactions. When a request arrives, SGLang performs a prefix match on the radix tree to locate existing cached KV tensors. Retained nodes remain in memory across requests and are evicted according to a least-recently-used (LRU) policy when GPU memory pressure rises.
This tree-structured approach enables fine-grained cache reuse across complex execution patterns:
- Multi-turn conversations: SGLang retains the full chat prefix in the radix tree, allowing subsequent turns to execute decode passes immediately without recomputing past turns.
- Branching reasoning trees: Workloads such as Monte Carlo tree search (MCTS) or majority-voting self-consistency reuse shared reasoning premises across parallel sampling branches.
- Retrieval-augmented generation (RAG): Repeated document context shared across distinct user queries is automatically indexed and reused without explicit manual cache pinning.
TensorRT-LLM employs an optimized paged KV memory structure directly coupled with NVIDIA's proprietary Fused Multi-Head Attention (FMHA) kernels, focusing primarily on minimizing memory access latency inside custom CUDA execution graphs.
Execution Graphs: Ahead-of-Time Compilation vs. Dynamic Runtimes
The architectural trade-off between TensorRT-LLM and its open-source dynamic counterparts centers on compilation lifecycle costs:
- TensorRT-LLM (Ahead-of-Time Compilation): TensorRT-LLM compiles model architectures into static TensorRT engines. It aggressively fuses layers (combining layer normalization, attention projections, and activation functions), auto-tunes General Matrix Multiply (GEMM) algorithms for specific NVIDIA GPU microarchitectures, and executes inference via pre-built CUDA graphs. This static optimization yields maximum hardware utilization but imposes a 20 to 30 minute engine compilation phase upon model initialization or quantization format changes.
- vLLM (Dynamic PyTorch and Custom CUDA Kernels): vLLM relies on dynamic execution using specialized CUDA and Triton kernels (such as FlashInfer, Marlin, and FlashAttention). It avoids lengthy ahead-of-time graph builds, achieving cold start initialization times under 60 seconds. The vLLM V1 architecture incorporates an asynchronous C++ execution scheduler that eliminates Python global interpreter lock (GIL) overhead in high-request environments.
- SGLang (Co-Designed Language and Runtime): SGLang pairs its RadixAttention runtime with a frontend interpreter and compiler. For structured outputs, SGLang utilizes XGrammar, which parses JSON schemas into compressed finite state machines. This avoids the token-by-token mask generation latency common in legacy regex decoders, producing valid structured outputs with minimal runtime penalty.
Throughput and Latency Benchmarks in Production
Framework performance varies substantially depending on request patterns, batch concurrency, and prefix homogeneity:
- Homogeneous Unique Requests: On isolated single-turn benchmarks with completely unique prompts, testing published by Spheron Network demonstrates that TensorRT-LLM delivers approximately 2,100 tokens per second at 50 concurrent requests, compared to 1,920 tokens per second on SGLang and 1,850 tokens per second on vLLM. TensorRT-LLM's compiled GEMM kernels and kernel fusions yield a 10% to 15% throughput advantage when cache reuse is unavailable.
- Shared-Prefix and Multi-Turn Workloads: When workloads contain repeated context (such as RAG document injection or interactive chat history), benchmarks reported by The AI Engineer show SGLang delivering up to 29% higher throughput (16,200 tokens per second versus 12,500 on vLLM on H100 clusters). By matching shared token sequences directly in the radix tree, SGLang bypasses redundant prefill computations entirely, drastically lowering Time-to-First-Token (TTFT).
- Mixture of Experts (MoE) Infrastructure: For massive MoE models such as DeepSeek-V3 and DeepSeek-R1, evaluations from Inference Engineering highlight SGLang's implementation of DeepEP (custom expert parallel communication kernels). SGLang sustains approximately 1,100 tokens per second on 8-GPU H100 clusters under FP8 quantization, scaling expert communication efficiently across multi-GPU nodes.
Operational Trade-Offs and Selection Framework
Selecting the appropriate serving runtime depends on deployment scale, hardware diversity, and traffic characteristics:
- Deploy TensorRT-LLM when: Serving high-volume, static-model APIs where traffic consists of independent single-turn requests; infrastructure is strictly standardized on modern NVIDIA hardware (H100, H200, B200); and operational deployment pipelines can accommodate 30-minute engine compilation stages.
- Deploy SGLang when: Workloads feature high prefix overlap, including agentic loops, multi-turn conversational bots, RAG pipelines with shared source documents, or high-throughput JSON schema extraction.
- Deploy vLLM when: Running multi-model inference fleets with frequent architecture updates, requiring rapid sub-minute container boot times, or operating across heterogeneous accelerator hardware including AMD ROCm, Intel Gaudi, and AWS Neuron.
Sources
- vLLM: Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.05519)
- SGLang: Efficient Execution of Structured Language Model Programs (arXiv:2312.07104)
- NVIDIA TensorRT-LLM Architecture Documentation (GitHub)
- Spheron Network: vLLM vs TensorRT-LLM vs SGLang Benchmarks
- The AI Engineer: vLLM vs Ollama vs SGLang vs TensorRT-LLM
- Inference Engineering: vLLM vs SGLang vs TensorRT-LLM Architecture Analysis
