inference129 articles

inference

Articles

Multi-Token Prediction (MTP): Mathematical Foundations, Shared Trunk Architectures, Sequential Future Verification, and Speculative Decoding Dynamics
The standard training objective for autoregressive large language models is next-token prediction (NTP), where model parameters $\theta$ are trained via maximum likelihood estimation to forecast a single subsequent token given all previous context. While this paradigm has driven modern foundation models, it enforces a myopic local optimization: the model learns transition probabilities strictly between adjacent tokens without explicit incentives to plan multi-step syntactic or semantic trajector
August 29, 20261 min
KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation
KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation In autoregressive transformer generation, memory consumption and serving throughput are dominated by the Key-Value (KV) cache. For long-context generation and continuous multi-turn dialogue, the linear growth of the KV cache with sequence length imposes an unsustainable memory footprint and saturates high-bandwidth GPU memory channels. Standar
August 28, 20261 min
Multi-Head Latent Attention (MLA): Mathematical Foundations, Low-Rank KV Compression, Decoupled Rotary Positional Embeddings, and Inference Matrix Absorption
Autoregressive transformer inference in production is fundamentally bounded by memory bandwidth rather than compute capability during the decoding phase. As context lengths scale from 8,000 to 128,000 tokens and beyond, storing and loading the Key-Value (KV) cache across tens of attention layers consumes the vast majority of GPU High-Bandwidth Memory (HBM) and degrades serving throughput. While earlier architectures mitigated this bottleneck via Multi-Query Attention (MQA) or Grouped-Query Atte
August 28, 20261 min
Disaggregated Prefill and Decode in Production LLM Serving: Comparing DistServe, Mooncake, Splitwise, and vLLM Architecture, KV Cache RDMA Transfer, TTFT/ITL Decoupling, and Cluster Economics
Disaggregated Prefill and Decode (PD Disaggregation) in Production LLM Serving: Comparing DistServe, Mooncake, Splitwise, and vLLM Architecture, KV Cache RDMA Transfer, TTFT/ITL Decoupling, and Cluster Economics In conventional large language model (LLM) inference engines, prompt processing (prefill) and autoregressive token generation (decode) execute on the same physical GPU workers. While colocated serving simplifies cluster orchestration, it creates a fundamental architectural contradiction
August 28, 20261 min
PagedAttention: Mathematical Foundations, Virtual Memory Block Allocation, Non-Contiguous KV-Cache Management, and Prefix Sharing Dynamics
PagedAttention is an attention algorithm and memory management architecture designed to eliminate memory waste and fragmentation in autoregressive Large Language Model (LLM) serving. Introduced by researchers at UC Berkeley and published at SOSP 2023 alongside the vLLM serving engine, PagedAttention adapts the classic operating system principles of virtual memory, paging, and copy-on-write to the management of dynamic Key-Value (KV) caches in GPU High Bandwidth Memory (HBM). During autoregressi
August 28, 20261 min
Intel Details 256-Core Xeon Diamond Rapids Architecture with 1.28GB L3 Cache and FP8 AMX
At the Hot Chips 2026 symposium, Intel unveiled the microarchitectural details for its next-generation data center platform, Xeon 7 (codenamed Diamond Rapids). Scheduled for production release in 2027, the flagship processor features up to 256 performance cores, 1.28GB of Last-Level Cache (LLC), and native FP8 support within its Advanced Matrix Extensions (AMX). The design marks a shift in Intel's high-performance server strategy, dropping Simultaneous Multi-Threading (SMT) entirely in favor of
August 28, 20261 min
LLM Inference Serving Engines in Production: Comparing vLLM, SGLang, TensorRT-LLM, and TGI Architecture, KV Cache Management, Chunked Prefill, and Serving Economics
Serving large language models in production environments requires managing two fundamentally different compute regimes: the compute-bound prefill phase and the memory-bandwidth-bound autoregressive decode phase. Over the past three years, the infrastructure stack for LLM inference has shifted from naive batched execution to specialized serving engines capable of continuous batching, non-contiguous key-value (KV) cache memory management, prefix caching, and compiler-level kernel fusion. Four sys
August 28, 20261 min
Speculative Decoding and Speculative Sampling: Mathematical Foundations, Lossless Rejection Sampling, Draft Model Architectures, and Serving Economics
Large language models generate text autoregressively, predicting one token at a time by passing the full sequence through dozens of transformer layers. During inference at low batch sizes, this generation process is fundamentally memory-bandwidth bound rather than compute bound. Speculative decoding addresses this bottleneck by using a fast drafting mechanism to propose candidate token sequences and verifying them in parallel with the primary target model in a single forward pass. When configur
August 28, 20261 min
Open-Source Embedding Models and Serving Frameworks in Production: Comparing BGE-M3, NV-Embed-v2, GTE-Qwen2, and ModernBERT-Embed Architecture, Matryoshka Projections, Context Scaling, and Serving Economics
Open-Source Embedding Models and Serving Frameworks in Production: Comparing BGE-M3, NV-Embed-v2, GTE-Qwen2, and ModernBERT-Embed Architecture, Matryoshka Projections, Context Scaling, and Serving Economics (TEI vs. Triton vs. vLLM) Text embeddings form the indexing and retrieval foundation for production Retrieval-Augmented Generation (RAG), semantic search, and agentic memory systems. While early production architectures relied almost exclusively on closed commercial APIs (such as OpenAI's te
August 28, 20261 min
Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, KV-Cache Bandwidth Reduction, Uptraining Recipes, and Tensor Parallelism Implications
Grouped-Query Attention (GQA) and Multi-Query Attention (MQA): Mathematical Foundations, KV-Cache Bandwidth Reduction, Uptraining Recipes, and Tensor Parallelism Implications The KV-Cache Bandwidth Wall Autoregressive decoder inference is bottlenecked by memory bandwidth, not compute. At each decoding step, the model must reload the entire key-value (KV) cache from high-bandwidth memory (HBM) into the compute units. For a model with $H$ attention heads, sequence length $n$, head dimension $d_
August 28, 20261 min
Local LLM Inference Frameworks in Production: Comparing llama.cpp, Ollama, Apple MLX, and Exo Distributed Clusters
The deployment landscape for large language models is bifurcating. While datacenter workloads rely on high-throughput continuous batching engines such as vLLM and TensorRT-LLM, local and edge deployments operate under fundamentally different physical constraints. On developer workstations, embedded hardware, and private office clusters, inference is rarely bound by compute saturation across thousands of concurrent requests. Instead, it is constrained by memory bandwidth, local VRAM capacity, hos
August 28, 20261 min
llama.cpp Merges DFlash 2 Support for Up to 2x Faster Speculative Decoding Across Long Contexts
The open-source llama.cpp project has merged native support for DFlash 2, bringing parallel speculative decoding and substantial inference throughput improvements to local LLM serving across CPU, Apple Silicon, and GPU backends. The implementation, integrated via Pull Request #27342, adds local convolution operators and candidate selector mechanics designed specifically for the DFlash 2 architecture. Non-Autoregressive Speculative Drafting Standard speculative decoding uses a smaller autoreg
August 27, 20261 min
PagedAttention and Virtual Memory Management: Mathematical Foundations of Non-Contiguous KV-Cache Allocation, Dynamic Block Translation, and Copy-on-Write Forking Mechanics
PagedAttention and Virtual Memory Management: Mathematical Foundations of Non-Contiguous KV-Cache Allocation, Dynamic Block Translation, and Copy-on-Write Forking Mechanics In autoregressive transformer serving, the generation phase is fundamentally constrained by GPU memory capacity and memory bandwidth rather than raw compute throughput. While the initial prefill phase processes the prompt in parallel with high arithmetic intensity, the subsequent token-by-token decoding phase computes attent
August 27, 20261 min
OpenAI Researcher Warns Ultrafast Inference Demands Autonomous Cyber Defense
Accelerating inference speeds in frontier artificial intelligence systems pose critical containment challenges that human security operators cannot manage in real time, according to OpenAI researcher roon. Commenting following the unveiling of custom inference hardware architectures and ultrafast model serving tiers, roon warned that unaligned or compromised agent systems running at 50 times baseline generation speeds could execute multi-stage network penetration and lateral movement before hum
August 27, 20261 min
OpenAI Details Custom Inference Chip 'Jalapeño' at Hot Chips, Targeting 700W Efficiency Against Nvidia Blackwell
OpenAI has revealed architectural specifications and benchmark data for its first in-house artificial intelligence accelerator, code-named Jalapeño. Presented by hardware lead Richard Ho at the Hot Chips conference at Stanford University, the application-specific integrated circuit (ASIC) is engineered specifically for large language model inference rather than model training. Developed over an 18-month partnership with Broadcom and manufactured by TSMC, the chip targets large-scale token gener
August 27, 20261 min
LLM Inference Engines in Production: Comparing vLLM, SGLang, TensorRT-LLM, and TGI
Production deployment of large language models requires balancing competing operational constraints: time-to-first-token (TTFT), inter-token latency (ITL), aggregate throughput, and VRAM utilization. Standard deep learning serving frameworks fail on autoregressive transformer inference because LLM workloads exhibit two distinct operational phases: the compute-bound prefill phase (processing the input prompt) and the memory-bandwidth-bound decode phase (generating tokens autoregressively one by o
August 27, 20261 min
StreamingLLM and Attention Sinks: Mathematical Foundations of Softmax Normalization Artifacts, Initial Token Anchoring, and Infinite-Context Rolling KV-Cache Mechanics
StreamingLLM and Attention Sinks: Mathematical Foundations of Softmax Normalization Artifacts, Initial Token Anchoring, and Infinite-Context Rolling KV-Cache Mechanics In autoregressive Large Language Models (LLMs), long-context deployment is constrained by the quadratic compute complexity of attention and the linear memory growth of the Key-Value (KV) cache. For an LLM processing a sequence of length $L$ with batch size $b$, hidden dimension $d$, and $n_{kv}$ key-value heads across $N_L$ trans
August 27, 20261 min
Multi-Head Latent Attention (MLA): Mathematical Foundations, Low-Rank KV Compression, Decoupled RoPE Mechanics, and Matrix Absorption
Multi-Head Latent Attention (MLA): Mathematical Foundations, Low-Rank KV Compression, Decoupled RoPE Mechanics, and Matrix Absorption Autoregressive inference in large language models is fundamentally constrained by memory bandwidth rather than floating-point computation throughput. During the generation phase, the transformer must load the Key-Value (KV) cache of all prior tokens from high-bandwidth memory (HBM) to on-chip SRAM for every newly generated token. As context lengths scale to hundr
August 27, 20261 min
Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration
Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration Serving modern large language models at scale requires addressing severe hardware constraints. In autoregressive generation, decoding is fundamentally bounded by memory bandwidth rather than floating-point computation throughput. Each generated token requires reading every parameter from High Bandwidth Memory (HBM) into SRAM to perfo
August 27, 20261 min
Speculative Decoding: Mathematical Foundations, Distribution Preservation Proofs, Tree-Structured Verification, and Memory-Bandwidth Amortization
Autoregressive large language model (LLM) generation suffers from an acute hardware efficiency mismatch during inference. While the prefill phase (processing the input prompt) processes tokens in parallel and achieves high arithmetic intensity on modern matrix accelerators, the decode phase (generating text token-by-token) is fundamentally memory-bandwidth bound. At small batch sizes, each generated token requires transferring the model's entire multi-billion-parameter weight matrix from High-Ba
August 27, 20261 min