Speculative Decoding in Production Serving: Comparing Small Draft Models, Medusa, EAGLE-2, and Lookahead Decoding Architecture, Verification Tree Overhead, and Throughput Economics

Large language model inference during autoregressive decoding is structurally memory-bandwidth bound. During generation, each forward pass loads the model weight matrices (tens to hundreds of gigabytes) from High-Bandwidth Memory (HBM) into on-chip SRAM to produce a single token. Because the arithmetic intensity is close to zero, modern accelerators like the NVIDIA H100 and B200 spend the vast majority of their compute cycles stalled on memory bus transfers rather than executing matrix multiplications.

Speculative decoding restructures this execution model. By using a lightweight draft mechanism to generate a sequence or tree of candidate tokens and verifying all candidates simultaneously in a single forward pass of the target model, speculative decoding converts memory-bound sequential token generation into compute-bound parallel verification. Understanding how different speculation architectures function, how tree attention verification is implemented, and where the batching crossover cliff occurs is critical for deploying high-throughput serving systems.

The Memory Bandwidth Wall and Speculative Verification

In standard autoregressive decoding, generating $K$ tokens requires $K$ sequential forward passes. When batch size is small (batch size 1 to 4), each forward pass reads all model parameters from HBM once while performing a single matrix-vector multiplication. For a 70B parameter model in FP16 (140 GB of weights), generating one token on an 80 GB H100 GPU (with 3.35 TB/s memory bandwidth) requires approximately 41 milliseconds of memory transfer time alone, regardless of the GPU's 989 TFLOPS of FP16 compute capacity.

Speculative decoding breaks this sequential dependency by pairing two distinct operations:

Draft Generation: A fast, low-cost draft mechanism proposes $\gamma$ candidate tokens $[x_1, x_2, \dots, x_\gamma]$ .
Target Verification: The primary target model executes a single forward pass across all $\gamma$ candidate positions simultaneously. Because verifying $\gamma$ tokens in parallel increases the batch dimension within matrix multiplications, arithmetic intensity increases by a factor of $\gamma$ , utilizing the previously idle Tensor Cores without increasing memory load duration.

Speculative decoding preserves the exact output distribution of the target model through modified rejection sampling, as formulated by Leviathan et al. (2022) and Chen et al. (2023). For each proposed token $x_i$ at step $i$ , let $p(x_i)$ denote the target model probability and $q(x_i)$ denote the draft model probability. The token is accepted with probability:

$\alpha_i = \min\left(1, \frac{p(x_i)}{q(x_i)}\right)$

If a token $x_i$ is rejected, the sampling loop stops at index $i$ , discards all subsequent drafted tokens $[x_{i+1}, \dots, x_\gamma]$ , and samples a replacement token from the normalized residual distribution:

$p_{\text{res}}(x) = \frac{\max(0, p(x) - q(x))}{\sum_{x'} \max(0, p(x') - q(x'))}$

This mathematical formulation guarantees that the resulting token sequence is provably identical in distribution to sampling directly from the target model $p(x)$ , ensuring zero quality degradation or perplexity drift.

Architectural Paradigms Compared

Serving frameworks implement speculative decoding using four primary architectural paradigms, each presenting different trade-offs in training complexity, VRAM consumption, and acceptance rates.

1. Separate Draft Models

The canonical approach uses an independent, smaller foundation model as the drafter (for example, pairing LLaMA-3.1-8B as the draft model for LLaMA-3.1-70B).

Mechanism: The draft model autoregressively generates a linear chain of $\gamma$ tokens. The target model then ingests the full candidate prefix in one batch verification pass.
Advantages: Requires no specialized training or architectural modifications; both models are standard causal transformers.
Engineering Bottlenecks: Both models must share an identical tokenizer vocabulary. The draft model requires dedicated GPU VRAM (for example, 16 GB for an 8B FP16 draft model), competing with the target model's KV cache allocation. Furthermore, orchestrating dual-model inference requires inter-model memory copies and synchronized tensor parallel schedules.

2. Medusa: Multi-Head Speculation

Introduced by Cai et al. (2024), Medusa eliminates the separate draft model by augmenting the primary model with $K$ feed-forward prediction heads attached to the final hidden layer of the frozen backbone.

Mechanism: Head $k$ predicts the token at position $t + k + 1$ directly from the hidden state at position $t$ . Instead of predicting a single linear sequence, Medusa outputs top- $k$ candidate tokens per head, assembling them into a tree of candidate sequences.
Advantages: Minimal VRAM footprint (only the parameters of $K$ lightweight MLP heads, typically under 2% of backbone parameters). Inference uses a single unified model runner.
Limitations: Because Medusa heads make predictions independently without conditioning on earlier draft tokens within the speculative window, feature uncertainty causes acceptance rates to drop steeply beyond the third speculative token ( $\gamma \ge 3$ ).

3. EAGLE and EAGLE-2: Feature-Level Extrapolation

Developed by Li et al. (2024a) and Li et al. (2024b), EAGLE (Extrapolation Algorithm for Greater Language-model Execution) resolves the feature uncertainty problem of multi-head decoding by operating at the hidden-state feature level.

Mechanism: EAGLE passes the top-layer hidden states from the target model alongside token embeddings into a single transformer decoder layer. It generates draft tokens autoregressively in feature space.
EAGLE-2 Dynamic Trees: While EAGLE-1 used a static draft tree topology, EAGLE-2 computes draft confidence scores at each expansion step to dynamically allocate tree branching. High-confidence paths are expanded deeper, whereas low-confidence branches are truncated early.
Performance: EAGLE-2 achieves per-token acceptance rates ( $\alpha$ ) between 70% and 85% across standard benchmarks, providing 2.5x to 3.5x wall-clock speedups over vanilla decoding while maintaining a parameter overhead below 1B parameters.

4. Training-Free and Native Speculation (Lookahead and MTP)

Lookahead Decoding: Formulated by Fu et al. (2024), Lookahead decoding treats autoregressive generation as solving a system of non-linear equations using Jacobi fixed-point iteration. It uses available GPU compute to simultaneously generate and verify multi-token n-grams without any external draft model or extra trained heads.
Prompt Lookup Decoding: As described by Saxena (2023), prompt lookup searches the existing prompt and generation context for recurring n-gram patterns, extracting candidates directly from input text. This approach achieves 60% to 80% acceptance rates on summarization, code editing, and retrieval-augmented workflows with zero parameter or training overhead.
Multi-Token Prediction (MTP): Pre-training architectures proposed by Gloeckle et al. (2024) and deployed in frontier models like DeepSeek-V3 integrate multi-token prediction heads directly into the foundational pre-training loss. This enables native speculative drafting during deployment without auxiliary draft models.

Verification Tree Overheads and KV Cache Management

Speculating a single linear sequence of $\gamma$ tokens has a failure mode: if the first drafted token $x_1$ is rejected, the remaining $\gamma - 1$ tokens are immediately discarded. Modern engines instead speculate candidate trees (e.g. 16 to 64 tree nodes evaluated concurrently).

Tree Attention Masks

To verify non-linear candidate trees in a single forward pass, the inference engine constructs a custom 2D causal tree attention mask. In this mask:

Each tree node $i$ can attend to all tokens in the historical prefix context.
Within the speculative tree, node $i$ can attend only to its direct ancestors in the tree hierarchy.
Sibling nodes and unrelated branches cannot attend to each other.

Custom FlashAttention and FlashInfer kernels execute this tree attention verification without materializing dense $N \times N$ attention matrices in GPU memory, bounding the verification latency close to that of a standard single-token prefill pass of length equal to the tree node count.

KV Cache Rollback Dynamics

Candidate tree exploration requires specialized memory management in engines like vLLM and SGLang:

Optimistic Allocation: The engine allocates physical KV cache blocks for all tree nodes in the candidate set prior to the target verification forward pass.
Path Selection and Pruning: After target verification computes logits across all tree positions, the engine identifies the longest accepted path from the root.
Rollback and Slot Reclaim: Physical memory blocks belonging to rejected tree branches must be reclaimed immediately. SGLang uses its RadixAttention tree structure to maintain token references, while vLLM uses PagedAttention block mapping to invalidate and recycle uncommitted physical pages without memory fragmentation.

The Batch Size Crossover Cliff and Serving Economics

While speculative decoding provides substantial speedups for single-stream generation, its economic profile shifts dramatically under concurrent production workloads.

Acceptance Rate and Expected Speedup

The theoretical speedup of speculative decoding is governed by the per-token acceptance rate $\alpha$ and the speculative window depth $\gamma$ . The expected accepted sequence length per step ( $\tau$ ) is:

$\tau = \sum_{i=1}^{\gamma} \alpha^i = \frac{\alpha(1 - \alpha^\gamma)}{1 - \alpha}$

Adding the guaranteed final token generated from the target distribution, the total progress per iteration is $1 + \tau$ . When $\alpha = 0.8$ and $\gamma = 5$ , the engine accepts on average 3.7 tokens per verification step, yielding a theoretical 3.7x step reduction.

The Batch Size Crossover Cliff

The effectiveness of speculative decoding depends on the underlying hardware bottleneck:

Low Concurrency (Batch Size 1 to 4): Execution is strictly memory-bandwidth bound. The target model forward pass on $\gamma$ tokens takes almost identical wall-clock time as a forward pass on 1 token. Speculative decoding delivers 2.0x to 3.5x end-to-end latency reduction.
High Concurrency (Batch Size 16 to 64+): As continuous batching packs dozens of concurrent requests into each forward pass, the aggregate sequence length naturally shifts the workload into the compute-bound regime (saturating GPU Tensor Cores).
Throughput Inversion: In compute-bound regimes, computing verification for rejected speculative tokens consumes arithmetic cycles that could otherwise process genuine user tokens. As a result, enabling speculative decoding at high batch concurrency degrades aggregate system throughput (tokens/second per GPU) by 15% to 40% compared to vanilla continuous batching, as detailed in production serving benchmarks by Spheron and BentoML.

Production Architectural Selection Matrix

Engineers deploying speculative decoding should evaluate techniques against their specific operational constraints:

Separate Small Draft Models: Best suited for off-the-shelf deployments where retraining is impossible. Requires strict tokenizer alignment and sufficient spare VRAM.
EAGLE-2 / EAGLE-3: Best suited for high-priority latency-sensitive endpoints (batch size 1 to 4) where training a 1-layer feature decoder is feasible. Delivers the highest acceptance rates (70% to 85%).
Medusa: Suitable for low-latency serving when feature-level training pipelines are unavailable and memory footprint must remain under 2% of backbone parameters.
Prompt Lookup / N-Gram: Optimal zero-cost choice for retrieval-augmented generation (RAG), document summarization, and code editing where prompt token repetition exceeds 50%.
Multi-Token Prediction (MTP): The target architecture for foundation model pre-training, eliminating auxiliary serving infrastructure entirely.

When designing production serving pipelines, speculative decoding should be configured with dynamic load-adaptive toggles: enabled during low-traffic periods to minimize Time-to-First-Token (TTFT) and Time-per-Output-Token (TPOT), and dynamically throttled as request queues grow to protect aggregate cluster throughput.