What Is Speculative Decoding?
Quick Answer
Speculative decoding pairs a small draft model with a larger target model. The draft model proposes several tokens ahead, and the target model verifies them all in one parallel forward pass instead of one sequential pass per token, which speeds up generation when the draft guesses are accepted.
- βΈA small draft model proposes multiple tokens ahead of the main model.
- βΈThe larger target model verifies the proposed tokens in a single parallel pass.
- βΈOutput quality matches the target model exactly β speculative decoding does not change what gets generated, only how fast.
Updated: July 14, 2026
Key Takeaways
- βA small draft model proposes tokens ahead; the larger target model verifies them in one parallel pass
- βOutput quality is identical to running the target model alone β only speed changes
- βSpeedup is largest on predictable text (code, repetitive structure) where the draft model guesses correctly more often
- βThe draft model must share the same tokenizer and vocabulary as the target model
How It Works
A draft model, much smaller and faster than the target model, generates a short sequence of candidate tokens. The target model then evaluates that entire sequence in a single forward pass, accepting tokens that match what it would have generated on its own and discarding the rest, along with anything generated after the first mismatch.
Because verifying several tokens in parallel costs roughly the same as generating one token sequentially on the target model, accepted guesses translate directly into a speed gain with no change to the final output. The target model always has final say β a wrong draft guess only costs the (small) time spent generating it, it never gets accepted into the output.
When Speculative Decoding Helps Most
Use speculative decoding when generating code, structured data, or other predictable text, where the draft model's guesses are accepted at a high rate and the speedup is largest. Use it when you are already inference-bound on generation speed rather than prompt-processing speed, since speculative decoding specifically accelerates the token-by-token generation step.
Skip speculative decoding when generating highly creative, unpredictable text (open-ended fiction, brainstorming), where the draft model's guesses are accepted less often and the speedup shrinks β the extra draft-model overhead can offset the gain in the worst case. Skip it if you don't have a compatible smaller model from the same family available to use as a draft model.
Setting Up a Draft Model
The draft model must share the same tokenizer and vocabulary as the target model β pairing models from different families (or even different sizes of a model where the vocabulary changed between versions) will not work. The most common setup uses a smaller model from the same family as the target (for example, a 1B-class model drafting for a 13B-class model from the same family).
Several local inference tools (including llama.cpp) support speculative decoding natively β the setup typically requires specifying both the target model and the draft model at launch, with no changes needed to how you prompt the model.