Speculative Decoding7 articles

Speculative Decoding

Articles

Speculative Decoding and Speculative Sampling: Mathematical Foundations, Lossless Rejection Sampling, Draft Model Architectures, and Serving Economics
Large language models generate text autoregressively, predicting one token at a time by passing the full sequence through dozens of transformer layers. During inference at low batch sizes, this generation process is fundamentally memory-bandwidth bound rather than compute bound. Speculative decoding addresses this bottleneck by using a fast drafting mechanism to propose candidate token sequences and verifying them in parallel with the primary target model in a single forward pass. When configur
August 28, 20261 min
llama.cpp Merges DFlash 2 Support for Up to 2x Faster Speculative Decoding Across Long Contexts
The open-source llama.cpp project has merged native support for DFlash 2, bringing parallel speculative decoding and substantial inference throughput improvements to local LLM serving across CPU, Apple Silicon, and GPU backends. The implementation, integrated via Pull Request #27342, adds local convolution operators and candidate selector mechanics designed specifically for the DFlash 2 architecture. Non-Autoregressive Speculative Drafting Standard speculative decoding uses a smaller autoreg
August 27, 20261 min
Speculative Decoding: Mathematical Foundations, Distribution Preservation Proofs, Tree-Structured Verification, and Memory-Bandwidth Amortization
Autoregressive large language model (LLM) generation suffers from an acute hardware efficiency mismatch during inference. While the prefill phase (processing the input prompt) processes tokens in parallel and achieves high arithmetic intensity on modern matrix accelerators, the decode phase (generating text token-by-token) is fundamentally memory-bandwidth bound. At small batch sizes, each generated token requires transferring the model's entire multi-billion-parameter weight matrix from High-Ba
August 27, 20261 min
Speculative Decoding: Mathematical Foundations, Exact Distribution Preservation via Rejection Sampling, and Tree-Structured Verification
Autoregressive decoding in large language models exhibits a severe computational asymmetry. Generating text token by token requires loading every parameter of a multi-billion-parameter network into high-bandwidth memory (HBM) for every forward pass. In low-batch or single-stream inference regimes, the compute units spend the majority of execution cycles waiting for memory transfers, yielding an arithmetic intensity of order O(1) FLOP per byte. Speculative decoding resolves this bottleneck by de
August 26, 20261 min
Speculative Decoding in Production Serving: Comparing Small Draft Models, Medusa, EAGLE-2, and Lookahead Decoding Architecture, Verification Tree Overhead, and Throughput Economics
Large language model inference during autoregressive decoding is structurally memory-bandwidth bound. During generation, each forward pass loads the model weight matrices (tens to hundreds of gigabytes) from High-Bandwidth Memory (HBM) into on-chip SRAM to produce a single token. Because the arithmetic intensity is close to zero, modern accelerators like the NVIDIA H100 and B200 spend the vast majority of their compute cycles stalled on memory bus transfers rather than executing matrix multiplic
August 25, 20261 min
Liquid AI Ships LFM2.5-DSpark Draft Models for Up to 3.2x Faster Inference
Liquid AI has released speculative decoding draft checkpoints for three models across its LFM2.5 series: LFM2.5-1.2B-Instruct, LFM2.5-2.6B, and the mixture-of-experts model LFM2.5-8B-A1B. The release introduces small companion models designed to accelerate auto-regressive generation without altering final token distributions. The draft models are available in Safetensors and GGUF formats on Hugging Face, with immediate support implemented for SGLang and llama.cpp. Architecture and Draft Desig
August 20, 20261 min
Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving
Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving Large language model inference is fundamentally constrained by memory bandwidth during the auto-regressive decoding phase. Because each token generation step requires loading billions of model parameters from high-bandwidth memory (HBM) to compute units for a single token, standard auto-regressive generation operates at low arithmetic intensity. Speculative decoding addresses this bottlene
August 19, 20261 min

Speculative Decoding

Articles

Speculative Decoding and Speculative Sampling: Mathematical Foundations, Lossless Rejection Sampling, Draft Model Architectures, and Serving Economics

llama.cpp Merges DFlash 2 Support for Up to 2x Faster Speculative Decoding Across Long Contexts

Speculative Decoding: Mathematical Foundations, Distribution Preservation Proofs, Tree-Structured Verification, and Memory-Bandwidth Amortization

Speculative Decoding: Mathematical Foundations, Exact Distribution Preservation via Rejection Sampling, and Tree-Structured Verification

Speculative Decoding in Production Serving: Comparing Small Draft Models, Medusa, EAGLE-2, and Lookahead Decoding Architecture, Verification Tree Overhead, and Throughput Economics

Liquid AI Ships LFM2.5-DSpark Draft Models for Up to 3.2x Faster Inference

Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving