llms.blog
Speculative Decoding7 articles

Speculative Decoding

Articles

  • Speculative Decoding and Speculative Sampling: Mathematical Foundations, Lossless Rejection Sampling, Draft Model Architectures, and Serving Economics

    Large language models generate text autoregressively, predicting one token at a time by passing the full sequence through dozens of transformer layers. During inference at low batch sizes, this generation process is fundamentally memory-bandwidth bound rather than compute bound. Speculative decoding addresses this bottleneck by using a fast drafting mechanism to propose candidate token sequences and verifying them in parallel with the primary target model in a single forward pass. When configur

    1 min

  • llama.cpp Merges DFlash 2 Support for Up to 2x Faster Speculative Decoding Across Long Contexts

    The open-source llama.cpp project has merged native support for DFlash 2, bringing parallel speculative decoding and substantial inference throughput improvements to local LLM serving across CPU, Apple Silicon, and GPU backends. The implementation, integrated via Pull Request #27342, adds local convolution operators and candidate selector mechanics designed specifically for the DFlash 2 architecture. Non-Autoregressive Speculative Drafting Standard speculative decoding uses a smaller autoreg

    1 min

  • Speculative Decoding: Mathematical Foundations, Distribution Preservation Proofs, Tree-Structured Verification, and Memory-Bandwidth Amortization

    Autoregressive large language model (LLM) generation suffers from an acute hardware efficiency mismatch during inference. While the prefill phase (processing the input prompt) processes tokens in parallel and achieves high arithmetic intensity on modern matrix accelerators, the decode phase (generating text token-by-token) is fundamentally memory-bandwidth bound. At small batch sizes, each generated token requires transferring the model's entire multi-billion-parameter weight matrix from High-Ba

    1 min

  • Speculative Decoding: Mathematical Foundations, Exact Distribution Preservation via Rejection Sampling, and Tree-Structured Verification

    Autoregressive decoding in large language models exhibits a severe computational asymmetry. Generating text token by token requires loading every parameter of a multi-billion-parameter network into high-bandwidth memory (HBM) for every forward pass. In low-batch or single-stream inference regimes, the compute units spend the majority of execution cycles waiting for memory transfers, yielding an arithmetic intensity of order O(1) FLOP per byte. Speculative decoding resolves this bottleneck by de

    1 min

  • Speculative Decoding in Production Serving: Comparing Small Draft Models, Medusa, EAGLE-2, and Lookahead Decoding Architecture, Verification Tree Overhead, and Throughput Economics

    Large language model inference during autoregressive decoding is structurally memory-bandwidth bound. During generation, each forward pass loads the model weight matrices (tens to hundreds of gigabytes) from High-Bandwidth Memory (HBM) into on-chip SRAM to produce a single token. Because the arithmetic intensity is close to zero, modern accelerators like the NVIDIA H100 and B200 spend the vast majority of their compute cycles stalled on memory bus transfers rather than executing matrix multiplic

    1 min

  • Liquid AI Ships LFM2.5-DSpark Draft Models for Up to 3.2x Faster Inference

    Liquid AI has released speculative decoding draft checkpoints for three models across its LFM2.5 series: LFM2.5-1.2B-Instruct, LFM2.5-2.6B, and the mixture-of-experts model LFM2.5-8B-A1B. The release introduces small companion models designed to accelerate auto-regressive generation without altering final token distributions. The draft models are available in Safetensors and GGUF formats on Hugging Face, with immediate support implemented for SGLang and llama.cpp. Architecture and Draft Desig

    1 min

  • Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving

    Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving Large language model inference is fundamentally constrained by memory bandwidth during the auto-regressive decoding phase. Because each token generation step requires loading billions of model parameters from high-bandwidth memory (HBM) to compute units for a single token, standard auto-regressive generation operates at low arithmetic intensity. Speculative decoding addresses this bottlene

    1 min