Speculative decoding

How it works

Decode is slow because the large model produces one token per step. In speculative decoding, a draft quickly guesses the next few tokens and the main model verifies all of them in a single forward pass, which costs about as much as one ordinary decode step because decode is limited by memory bandwidth. Accepted tokens are kept; at the first rejection the main model's own token is used and drafting starts again.

With the standard acceptance rule, the output follows the same distribution as the main model alone, so quality is unchanged and only speed differs. Drafts can be a smaller model from the same family, extra heads trained on the main model, or n-gram lookup from the prompt. The speedup depends on how often drafts are accepted, so predictable text such as code edits gains most.

API users see it only as speed, though some APIs let you pass the text you expect back (for example a file being edited), which works on the same principle. When self-hosting, engines such as vLLM, SGLang, TensorRT-LLM and llama.cpp support it; the draft costs extra memory and compute, and the gain shrinks at large batch sizes, where the GPU is already busy.

Worked example

Illustrative: the draft proposes 4 tokens per step and on average 3 are accepted, so each verification pass yields 3 accepted tokens plus 1 from the main model, 4 tokens instead of 1. If drafting plus verification takes 1.5 times as long as a normal decode step, output speed rises about 4 / 1.5 = 2.7 times.

Common mistakes

• 1Expecting the same speedup at high concurrency, where verification work competes with other requests for a saturated GPU.
• 2Pairing a draft model whose tokenizer differs from the main model's, which most implementations don't support.
• 3Judging the gain on one kind of prompt; acceptance rates differ widely between code, chat and creative text.