Speculative Decoding

What is Speculative Decoding?

Speculative decoding is an inference optimization technique that accelerates text generation in large language models without altering the output distribution. In standard autoregressive generation, a model produces tokens one at a time, sequentially predicting the next most likely token based on the previous context. This process is constrained by memory bandwidth rather than compute. Modern GPUs spend more time loading model weights from memory into compute cores than they spend performing calculations, which leaves expensive hardware underutilized.

Speculative decoding addresses this bottleneck with a dual-model architecture: a much smaller, faster "draft" model and the original, larger "target" model. Drafting several candidate tokens with the small model is cheap, and verifying those tokens in parallel with the large model takes roughly the same time as generating a single token in isolation. By speculatively generating a sequence of tokens and verifying them in one forward pass, the technique can yield multiple validated tokens per step, increasing the token generation rate.

The technique is lossless: unlike quantization, weight pruning, or other compression methods that alter model weights and can degrade output quality, speculative decoding guarantees the final output matches exactly what the large model would have produced on its own. It is an algorithmic acceleration method that uses the parallel processing capabilities of modern accelerators to work around the sequential nature of language modeling.

How Speculative Decoding Works (The Draft-and-Verify Cycle)

The mechanism behind speculative decoding is a continuous, iterative cycle of drafting and verifying. This works because for many simple, structural, or highly predictable phrases, a small model predicts the next words about as well as a much larger one.

1. The Drafting Phase

In the initial step of the cycle, the smaller draft model (often 10x to 50x smaller than the target model) autoregressively generates a sequence of K candidate tokens. For instance, if the prompt is "The capital of France is", the draft model might rapidly generate the tokens [" Paris", ",", " which", " is"]. Because the draft model has far fewer parameters, it can generate these K tokens in a fraction of the time it would take the large model to generate a single token.

2. The Verification Phase

Once the K candidate tokens are drafted, they are appended to the original context and passed to the large target model. The target model processes this entire sequence in a single parallel forward pass. Modern GPUs handle parallel computation across a sequence efficiently, so calculating the probabilities for these K tokens simultaneously takes nearly the same time as calculating the probability for one single token.

The target model evaluates the probabilities of the draft model's predictions. It checks whether the token generated by the draft model would also have been sampled by the target model under its own probability distribution.

3. The Acceptance/Rejection Step

The algorithm then sequentially compares the draft tokens against the target model's distribution. As long as the draft tokens align with what the target model would produce, they are accepted into the final output sequence.

• If a token is accepted, the process moves to evaluate the next token in the drafted sequence.
• The moment a token is rejected (meaning the large model disagrees with the small model's prediction), the drafted sequence is truncated at that point. The target model then provides the correct replacement token for the rejected one, so the sequence progresses correctly.
• Even in the worst case, where all drafted tokens are immediately rejected, the target model still produces one valid token, so the system never falls below the baseline speed of standard autoregressive generation, minus the small overhead of running the draft model.

Mathematical Guarantee: Lossless Sampling & Acceptance Rates

Speculative decoding carries a mathematical guarantee of lossless sampling. The algorithm uses a technique derived from rejection sampling to ensure the probability distribution of the accepted tokens matches the target model's original distribution exactly.

Call p(x) the probability the target model assigns to token x, and q(x) the probability the draft model assigns to it. When the draft model proposes token x, the target model accepts it with probability min(1, p(x)/q(x)): if the target model considers the token at least as likely as the draft model does, it is accepted outright; otherwise it is accepted with probability equal to that ratio.

If the token is rejected, a new token is resampled from a modified distribution built from max(0, p(x) − q(x)) for each possible token, normalized so the values sum to 1. This construction ensures the overall expected distribution of generated tokens is exactly p(x). Whether the temperature is set to 0 (greedy decoding) or a higher value for creative sampling, the final outputs remain mathematically indistinguishable from those generated solely by the target model.

The efficiency of speculative decoding depends heavily on the acceptance rate, the percentage of draft tokens the target model approves. A higher acceptance rate translates to a higher generation speedup, though the exact relationship depends on draft length K, the size gap between draft and target models, and the hardware running the verification pass. As a rough illustration, holding K around 5 tokens per step, low acceptance rates (around 20%) tend to yield only modest speedups (roughly 1.1x), while high acceptance rates (around 80-90%) can approach 3x or more. Real deployments should benchmark on their own model pair and hardware rather than rely on generic figures.

Variants: Draft Models, EAGLE, Medusa, and Model-Free Methods

While standard speculative decoding requires two distinct models, the field has developed alternative approaches that achieve similar or better acceleration without the operational overhead of maintaining a separate draft model.

EAGLE and its successors have largely displaced separate draft models where a trained speculator exists. Rather than running a second full model, EAGLE attaches a small draft head that reads the target model's own hidden states; EAGLE-3 (March 2025) fuses low-, mid-, and high-level features and predicts tokens directly, reporting speedups up to 6.5x in the paper's benchmarks, roughly 1.4x better than EAGLE-2. Production numbers are lower but still substantial: Red Hat's July 2025 vLLM benchmarks measured up to 1.8x lower latency for Llama 3.1 8B and up to 1.6x for Llama 3.3 70B with EAGLE-3 speculators on A100s at low request rates, with up to 2.1x on RAG and math workloads.

A related approach is multi-token prediction (MTP), where the target model itself is trained to predict several future tokens per step. DeepSeek-V3 ships native MTP heads, and inference engines can reuse them as a built-in speculator with no extra model at all.

Model-free methods trade lower ceilings for zero setup. N-gram (prompt lookup) drafting proposes tokens by matching strings that already appear in the context, which works well when outputs copy from inputs, as in RAG, summarization, and code editing. Suffix decoding generalizes this by matching repeated suffixes across prior generations with dynamic speculation depth. Medusa appends multiple decoding heads to the target model to predict several future tokens simultaneously, and lookahead decoding drafts via parallel n-gram paths without any extra parameters.

Hardware & Production Infrastructure

Deploying speculative decoding in production requires inference infrastructure capable of managing complex memory states and orchestrating two neural networks simultaneously.

Memory Requirements

Speculative decoding increases VRAM requirements. The GPUs must hold the parameters and KV cache of the large target model as well as the parameters and KV cache of the draft model. Because the draft model generates K tokens per step, the KV cache grows in bursts, requiring dynamic memory management systems like PagedAttention to prevent fragmentation and out-of-memory errors during peak loads.

Production Frameworks

Leading open-source inference engines have built-in support for speculative decoding:

• vLLM: As of 2026, vLLM's speculative decoding support covers EAGLE/EAGLE-3, MTP, separate draft models, n-gram, and suffix decoding, all configured through a single --speculative-config flag, while handling the orchestration of continuous batching and KV caches between the speculator and target model. Since v0.9.1 it also exposes live acceptance-rate and mean-acceptance-length metrics so operators can verify a speculator is actually paying for itself.
• SGLang: Ships EAGLE-2 and EAGLE-3 speculative decoding integrated with its RadixAttention prefix cache, and is a common choice for serving DeepSeek models with their native MTP heads as the speculator.
• TensorRT-LLM: NVIDIA's inference engine offers highly optimized speculative decoding (draft-target, EAGLE, Medusa, and lookahead), using custom CUDA kernels to minimize the latency overhead of the draft-and-verify cycle and maximize hardware utilization.

Managed APIs productize the same idea. OpenAI's Predicted Outputs feature (launched November 2024) lets callers pass an expected completion, such as a file being edited, which the server uses as the draft sequence, cutting latency for code-editing workloads without any client-side model management.

Optimal hardware for speculative decoding involves high-bandwidth memory to rapidly feed the target model during the verification phase. Since the target model is memory-bound during verification, the faster it can read its weights to verify the draft sequence, the greater the overall speedup, whether the deployment runs on current-generation NVIDIA accelerators or other high-bandwidth-memory hardware.

Frequently Asked Questions

Does speculative decoding degrade model quality or hallucinate more? No. Standard speculative decoding is mathematically proven to be strictly lossless. The final output distribution is exactly the same as if the large target model generated the text alone. It is purely a speed optimization and has no impact on reasoning, hallucinations, or output quality.

What makes a good draft model for production use? A good draft model must be significantly smaller (to be fast) but heavily aligned with the target model (to have a high acceptance rate). Models from the same family often work best, for example a small Llama 4 variant drafting for a larger Llama 4 model, because they share tokenizers, vocabularies, and internal structural representations of language.

Why not just use the small draft model if it is so accurate? While a small model might predict the next word correctly 70% of the time for simple grammar or structural tokens, the 30% of the time it gets it wrong usually involves complex reasoning, factual accuracy, or nuanced instruction following. Speculative decoding relies on the larger target model, often a transformer-based architecture, to catch those errors and provide the reasoning quality only a frontier model can offer.

Why do speculative decoding speedups shrink at high batch sizes? Speculative decoding converts spare compute into speed: at low batch sizes the GPU is memory-bound, so verifying K tokens in one pass is nearly free. As continuous batching fills the GPU with concurrent requests, decoding becomes compute-bound, the "free" parallel verification now competes with other requests, and rejected draft tokens turn into wasted FLOPs. Red Hat's 2025 vLLM benchmarks showed EAGLE-3 gains at low request rates degrading, and sometimes inverting, at high request rates. This is why latency-sensitive, low-concurrency serving benefits most, and why lightweight methods like n-gram drafting are preferred when traffic is spiky.

Which speculative decoding method should I pick? If a trained EAGLE-3 speculator exists for your model (or the model ships MTP heads, as DeepSeek-V3 does), use it: trained feature-level speculators deliver the highest acceptance rates. If not, and your outputs frequently copy from the input (RAG, summarization, code edits), n-gram or suffix decoding gives a modest speedup for zero setup cost. Reach for a separate draft model only when both options are unavailable and a well-aligned smaller model from the same family exists. Whatever you choose, benchmark acceptance rate on your own traffic; a speculator with a low acceptance rate adds drafting overhead without recovering it in accepted tokens.

Does speculative decoding improve Time-to-First-Token (TTFT)? No. Speculative decoding only accelerates the generation phase (time between tokens, or decoding latency). The initial prompt processing (the prefill phase) is still handled entirely by the large target model sequentially, so TTFT remains unchanged. This differs from test-time-compute techniques, which trade additional inference time for improved output quality rather than raw speed.