vLLM7 articles

vLLM

Articles

LLM Inference Serving Engines in Production: Comparing vLLM, SGLang, TensorRT-LLM, and TGI Architecture, KV Cache Management, Chunked Prefill, and Serving Economics
Serving large language models in production environments requires managing two fundamentally different compute regimes: the compute-bound prefill phase and the memory-bandwidth-bound autoregressive decode phase. Over the past three years, the infrastructure stack for LLM inference has shifted from naive batched execution to specialized serving engines capable of continuous batching, non-contiguous key-value (KV) cache memory management, prefix caching, and compiler-level kernel fusion. Four sys
August 28, 20261 min
LLM Inference Engines in Production: Comparing vLLM, SGLang, TensorRT-LLM, and TGI
Production deployment of large language models requires balancing competing operational constraints: time-to-first-token (TTFT), inter-token latency (ITL), aggregate throughput, and VRAM utilization. Standard deep learning serving frameworks fail on autoregressive transformer inference because LLM workloads exhibit two distinct operational phases: the compute-bound prefill phase (processing the input prompt) and the memory-bandwidth-bound decode phase (generating tokens autoregressively one by o
August 27, 20261 min
CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination
CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination In high-throughput large language model serving, the autoregressive generation (decode) phase presents an operational bottleneck distinct from prefill processing. While prefill computation is dominated by compute-bound General Matrix Multiplications (GEMMs) operating across extended sequence lengths, autoregressive decoding processes a single token per sequence at each ite
August 22, 20261 min
Netflix Details GenRec LLM-Native Recommendation Architecture in Production A/B Trials
Netflix has detailed GenRec, an internal machine learning architecture that adapts open-weight large language models for production recommendation ranking. The system replaces hand-crafted feature pipelines with natural-language context engineering, achieving measurable improvements in live A/B trials while reducing required training labels by up to 40 times. For years, industrial recommendation engines at scale have depended on complex feature stores tracking thousands of engineered numerical
August 22, 20261 min
LLM Autoscaling and Cold Starts in Kubernetes: Architecture, KEDA Metrics, Model Weight Caching, and Ephemeral GPU Provisioning
Autoscaling large language model workloads on Kubernetes presents a fundamentally different engineering problem than traditional stateless microservices. While web APIs scale on CPU utilization or request rate within seconds, LLM inference instances require specialized GPU accelerators, massive container images, multi-gigabyte weight tensors, and intensive runtime compilation before serving a single token. Without proactive architectural design, a cold-starting LLM pod on Kubernetes often requi
August 20, 20261 min
Benchmarking LLM Inference in Production: Architecture, Metrics, and Tooling Across AIPerf, GuideLLM, and LLMPerf
Traditional HTTP load-testing tools such as Apache Bench, wrk, and Locust evaluate systems using uniform request-response cycles. These tools send a payload, wait for the full response, and compute metrics such as requests per second (RPS) and round-trip latency percentiles. For stateless REST APIs, this model aligns directly with user experience. Large language model (LLM) serving fundamentally breaks this abstraction. An inference request is not an atomic computation. It consists of two disti
August 19, 20261 min
LLM Serving Frameworks in Production: SGLang vs. vLLM vs. TensorRT-LLM Architecture, Memory Management, and Throughput Trade-Offs
Serving large language models in production has moved beyond simple iteration-level scheduling. While basic continuous batching solved the fundamental problem of GPU starvation during autoregressive decoding, modern production demands have shifted the engineering bottleneck toward KV cache memory reuse, compilation overhead, and structured generation efficiency. Three dominant serving runtimes currently lead high-throughput deployments: vLLM, SGLang, and NVIDIA TensorRT-LLM. Each framework appr
August 18, 20261 min

vLLM

Articles

LLM Inference Serving Engines in Production: Comparing vLLM, SGLang, TensorRT-LLM, and TGI Architecture, KV Cache Management, Chunked Prefill, and Serving Economics

LLM Inference Engines in Production: Comparing vLLM, SGLang, TensorRT-LLM, and TGI

CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination

Netflix Details GenRec LLM-Native Recommendation Architecture in Production A/B Trials

LLM Autoscaling and Cold Starts in Kubernetes: Architecture, KEDA Metrics, Model Weight Caching, and Ephemeral GPU Provisioning

Benchmarking LLM Inference in Production: Architecture, Metrics, and Tooling Across AIPerf, GuideLLM, and LLMPerf

LLM Serving Frameworks in Production: SGLang vs. vLLM vs. TensorRT-LLM Architecture, Memory Management, and Throughput Trade-Offs