A language model ordinarily generates output one token at a time, conditioned on the tokens it has already produced. Speculative decoding drafts several candidates and has the target model verify them together. If the draft is useful, one verification round can commit several tokens. If most candidates are rejected, drafting can add work without paying for itself.
The 2026 vLLM and NVIDIA NIM documentation describes multiple approaches, including separate draft models and prediction heads built into a model. Headline speedups only make sense alongside the target model, implementation and concurrent load in the test.
Trace a verification round
Suppose a draft proposes four tokens and the target accepts the first two. At the first rejection, the implementation applies its correction or sampling step and discards the unused continuation. Some methods can also produce an extra token after accepting a full draft. The speed comparison should count committed output, rather than simply counting the number of candidates.
An exact speculative sampling scheme can preserve the target model’s output distribution. That is a statement about probabilities. With stochastic generation, it does not mean two individual runs will print identical text. Check what the implementation promises under the settings you use.

Advertisement
Find the break-even point
Suppose ordinary decoding takes 12 ms per output token. If drafting and verification together take 24 ms per round, committing an average of three output tokens gives 8 ms per token. Committing 1.5 gives 16 ms per token, slower than the baseline. These are illustrative assumptions, and committed output includes any correction the round actually emits.
Longer drafts can add verification work, so a higher draft length is not an automatic win. Compare output committed per round with the total time for that round. Keep the baseline and workload fixed when testing a change.

Separate first-token latency, generation speed and throughput
Time to first token includes work before the first output, such as handling the prompt and queueing. Inter-token latency measures gaps during output. Server throughput aggregates output across requests. These metrics can move differently as concurrent demand grows. A draft model or auxiliary state can also consume memory that would otherwise be available for batching or cache.
← Swipe to view the full table →
| Item | What to verify |
|---|---|
| First output | Prompt handling and queueing affect TTFT |
| During generation | Time between output tokens |
| Across the server | Aggregate output and memory under concurrency |
Advertisement
Make benchmark results comparable
Record target model, drafting method, input and output lengths, concurrent request count and the metric being reported. Keep an eye on memory use and slow responses under load. Compare a short structured response and a longer free-form response separately before using one speed estimate for both workloads.

Is speculative decoding the same as a KV cache?
Speculative decoding changes how candidate output is generated and verified. A KV cache retains attention state from earlier tokens. A serving system can use both, but each solves a different problem.
Related reading: What the KV cache stores during inference
Sources and review date
- vLLMの投機的デコード文書
- NVIDIA NIMの対応と注意点
- Leviathan et al.: Fast Inference from Transformers via Speculative Decoding
Sources reviewed November 12, 2026. Figures and the cover were created for this article. Dated disclosures and illustrative examples are labeled in the figures.


Comments