Speculative Decoding Explained: When Drafting Tokens Actually Speeds Up LLMs

Speculative decoding。When drafting makes generation faster

A language model ordinarily generates output one token at a time, conditioned on the tokens it has already produced. Speculative decoding drafts several candidates and has the target model verify them together. If the draft is useful, one verification round can commit several tokens. If most candidates are rejected, drafting can add work without paying for itself.

The 2026 vLLM and NVIDIA NIM documentation describes multiple approaches, including separate draft models and prediction heads built into a model. Headline speedups only make sense alongside the target model, implementation and concurrent load in the test.

Advertisement

TOC

Trace a verification round

Suppose a draft proposes four tokens and the target accepts the first two. At the first rejection, the implementation applies its correction or sampling step and discards the unused continuation. Some methods can also produce an extra token after accepting a full draft. The speed comparison should count committed output, rather than simply counting the number of candidates.

An exact speculative sampling scheme can preserve the target model’s output distribution. That is a statement about probabilities. With stochastic generation, it does not mean two individual runs will print identical text. Check what the implementation promises under the settings you use.

Draft, verify and commit。Draft Propose candidate tokens 1, 2, 3 and 4 / Verify Accept 1 and 2, correct at 3, discard 4 / Continue Use the committed output in the next round
Illustrative round with two accepted draft tokens

Advertisement

Find the break-even point

Suppose ordinary decoding takes 12 ms per output token. If drafting and verification together take 24 ms per round, committing an average of three output tokens gives 8 ms per token. Committing 1.5 gives 16 ms per token, slower than the baseline. These are illustrative assumptions, and committed output includes any correction the round actually emits.

Longer drafts can add verification work, so a higher draft length is not an automatic win. Compare output committed per round with the total time for that round. Keep the baseline and workload fixed when testing a change.

How much output fits in a 24 ms round?。3 tokens committed on average 8 ms per token / 2 tokens committed on average 12 ms per token: baseline / 1.5 tokens committed on average 16 ms per token
Illustrative baseline: 12 ms per output token

Separate first-token latency, generation speed and throughput

Time to first token includes work before the first output, such as handling the prompt and queueing. Inter-token latency measures gaps during output. Server throughput aggregates output across requests. These metrics can move differently as concurrent demand grows. A draft model or auxiliary state can also consume memory that would otherwise be available for batching or cache.

← Swipe to view the full table →

Generation performance measurements
ItemWhat to verify
First outputPrompt handling and queueing affect TTFT
During generationTime between output tokens
Across the serverAggregate output and memory under concurrency

Advertisement

Make benchmark results comparable

Record target model, drafting method, input and output lengths, concurrent request count and the metric being reported. Keep an eye on memory use and slow responses under load. Compare a short structured response and a longer free-form response separately before using one speed estimate for both workloads.

Measure three kinds of speed。First output Prompt handling and queueing affect TTFT / During generation Time between output tokens / Across the server Aggregate output and memory under concurrency
Match the workload and measurement scope

Is speculative decoding the same as a KV cache?

Speculative decoding changes how candidate output is generated and verified. A KV cache retains attention state from earlier tokens. A serving system can use both, but each solves a different problem.

Related reading: What the KV cache stores during inference

Sources and review date

Sources reviewed November 12, 2026. Figures and the cover were created for this article. Dated disclosures and illustrative examples are labeled in the figures.

Back to the English home page

Share this article

Advertisement

Comments

To comment

five × three =

TOC