LLM Inference4 articles

LLM Inference

Articles

FP8 Mixed-Precision Serving in Production: Comparing E4M3 vs. E5M2 Formats, Per-Tensor vs. Block-Wise Dynamic Scaling, FP8 KV Cache, and Tensor Core GEMM Economics
FP8 Mixed-Precision Serving in Production: Comparing E4M3 vs. E5M2 Formats, Per-Tensor vs. Block-Wise Dynamic Scaling, FP8 KV Cache, and Tensor Core GEMM Economics Serving large language models at enterprise scale requires balancing computational throughput against high-bandwidth memory (HBM) capacity. While 16-bit floating-point formats (FP16 and BF16) remain the standard for model pre-training and fine-tuning, their memory footprint and arithmetic bandwidth create severe bottlenecks during hi
August 26, 20261 min
Disaggregated Prefill and Decode in Production LLM Serving: Architecture, Network KV Cache Migration, Chunked Prefill Trade-Offs, and Asymmetric Hardware Economics
Large language model serving systems have historically treated transformer execution as a homogeneous sequence of forward passes over a single unified GPU pool. Under continuous batching engines, incoming requests execute their prompt evaluation (prefill) and autoregressive token generation (decode) on the exact same accelerators, co-locating both phases within shared iteration batches. While continuous batching improves GPU compute utilization compared to static batching, co-locating prefill a
August 25, 20261 min
Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck
Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ
August 19, 20261 min
Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving
Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving Large language model inference is fundamentally constrained by memory bandwidth during the auto-regressive decoding phase. Because each token generation step requires loading billions of model parameters from high-bandwidth memory (HBM) to compute units for a single token, standard auto-regressive generation operates at low arithmetic intensity. Speculative decoding addresses this bottlene
August 19, 20261 min

LLM Inference

FP8 Mixed-Precision Serving in Production: Comparing E4M3 vs. E5M2 Formats, Per-Tensor vs. Block-Wise Dynamic Scaling, FP8 KV Cache, and Tensor Core GEMM Economics

Disaggregated Prefill and Decode in Production LLM Serving: Architecture, Network KV Cache Migration, Chunked Prefill Trade-Offs, and Asymmetric Hardware Economics

Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck

Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving