NOTES / INFERENCE AND SERVING

Speculative decoding

A small draft model proposes several tokens and the large model verifies them in one pass, giving faster decoding without changing the output distribution.

UPDATED 18 NOV 2026#inference#speculative-decoding#latency#decoding

The bottleneck

Generating K tokens normally takes K serial runs of the large model, because each token depends on the one before it1. Each run reads all the weights from memory to produce one token, which wastes much of the GPU’s compute (Latency and cost).

The trick

Two research groups described the same idea independently. Use a cheaper draft model to guess several next tokens, then have the large target model check all the guesses in a single parallel pass12. Leviathan and colleagues rest the idea on two observations: hard text-generation tasks contain easier parts that a smaller model approximates well, and a large model can score several tokens in parallel for about the cost of one step1.

Diagram: a draft model proposes five tokens, the target model scores them in one pass, and the accepted prefix is kept followed by the target's own token

Diagram: my own.

Steps:

  1. The draft model proposes a short run of tokens.
  2. The target model computes its own probabilities for those positions in one pass.
  3. A modified rejection-sampling rule accepts a prefix of the proposed tokens and, at the first rejection, replaces that token with one sampled from the target’s distribution2.
  4. Repeat from the new position.

Every round yields at least one target-quality token, and sometimes several.

What it guarantees

Both papers stress that the output distribution is unchanged. Leviathan et al. state the method samples from the large model without changing the outputs, using no retraining or architecture change1. DeepMind’s version preserves the target model’s distribution, subject to hardware numerics, and reported a 2 to 2.5 times decoding speedup on a 70-billion-parameter Chinchilla model in a distributed setup2. That speedup is specific to their model, hardware and tasks.

The “same distribution” claim is about sampling. With greedy decoding the results should match what the target model would produce, up to numerical differences.

What decides the speedup

  • Acceptance rate. If the draft often guesses what the target would say, many tokens are accepted per pass. Predictable text, such as code boilerplate, quotations or repeated structure, accepts more than creative text.
  • Draft cost. The draft must be much cheaper than the target, or the gains vanish.
  • Load. The method spends spare compute to save memory-bound steps. When a server is already busy with large batches, there is less spare compute, so the benefit shrinks (my understanding; test it at your own concurrency).
  • Extra memory. The draft model needs space too.

Variants

Engines now offer several ways to get drafts. vLLM’s repository documentation lists speculative decoding with options such as n-gram lookup, suffix methods and EAGLE-style draft heads3. Check the engine’s current docs, since this is a fast-moving area.

When to use it

  • You self-host a large model for interactive use, and decode speed is the problem.
  • Outputs are long and fairly predictable.
  • You can measure acceptance rate and latency on real traffic.

It does not help prefill, so long prompts with short answers see little gain. If you use a hosted API, any such acceleration is the provider’s concern, though you may see it described as a faster output mode. Combine it with other levers: quantisation shrinks the weights that each step reads, and batching raises throughput.

Sources

  1. [1]Fast Inference from Transformers via Speculative Decoding — Leviathan, Kalman, Matias · 2023PARAPHRASED
  2. [2]Accelerating Large Language Model Decoding with Speculative Sampling — Chen et al., DeepMind · 2023PARAPHRASED
  3. [3]vLLM: High-Throughput LLM Inference and Serving Engine (repository overview) — vLLM project, via docsearchFEATURE LIST, AS SUMMARISED BY A THIRD PARTY

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.