Speculative Drafting · S01
Speculative decoding
Guess cheaply, check in one pass, keep only what the big model would have written.
The M and P tracks build and serve dLLMs. This track goes the other way: it uses ideas from diffusion to serve an AR model faster. A dLLM on its own writes worse than an AR model of the same size (M06, M12). But it can guess a whole block in one pass, and in speculative decoding a guess only has to be right often, because every guess is checked.
Draft, verify, keep
A cheap drafter proposes tokens. The target, the AR model we want to serve, reads them all in one forward pass, which gives its own choice at every position. It keeps the longest prefix where the draft matches its choices, then adds one token of its own at the first mismatch. The output is exactly what the target would have written alone; only the speed changes.
Why checking is cheap: at batch 1 a forward pass is limited by reading the weights, not by arithmetic (P08). Reading 5 positions costs about the same as reading 1.
With sampling, matching is replaced by a coin flip. A draft token with drafter probability and target probability is kept with probability . On a rejection, the target samples from what is left, , renormalized. The output then has exactly the target’s distribution.
What a drafter has to do
With the tokens committed per round, accepted drafts plus the target’s own one, the time per token is
So a drafter helps by being cheap (small ) or accurate (large ). The two pull against each other. An AR drafter pays one pass per guess, so it must stay small, and a small drafter is accepted less. Diffusion offers a way out: a block of guesses for one pass.
| Who drafts | How | Draft cost | Note | |
|---|---|---|---|---|
| AR drafter (Medusa, EAGLE-3, MTP) | small heads or layers on the target’s features | one token per pass, left to right | grows with | S02 |
| Diffusion drafter (DFlash 1 and 2) | a separate small block-diffusion model | a whole block in one pass | about one pass, whatever | S03 |
| Semi-AR drafter (DSpark) | in between | a few tokens per pass | S04 | |
| Self-speculative (Sigma, NLD) | the same model’s diffusion head | a whole block in one pass | one pass of the big model | S06 |
S05 checks a tree of guesses instead of a single chain. S07 looks at what happens with many requests at once, where verification stops being free.
References
- Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
- Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling. 2023.