Paper notes · arXiv 2025
Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing (D2F)
Block diffusion (M08) gives an exact KV cache, but blocks run one after another. D2F keeps the block-causal mask and adds inter-block parallelism: the next block starts while the previous one is still being decoded. Two parts:
- Training (§4.1–4.2): turn LLaDA or Dream into a block-causal model by distillation. The new part is that each block learns to denoise from partly masked earlier blocks, not clean ones. That is the “forcing”.
- Inference (§4.3): a pipeline of active blocks. A new block is appended once the last one is partly decoded, and it becomes fully active once its predecessor is nearly done.
What “forcing” means. In teacher forcing, the usual way to train AR models and block diffusion, each block is fed a clean prefix. Diffusion forcing (Chen et al., 2024, from video) feeds a noised one: masks rise from left to right, so earlier blocks are partly masked, but less than the current one. That is exactly what the pipeline looks like at inference, so the next block can start before the previous one is finished. A teacher-forced model has never seen an unfinished prefix, so it has to wait.
Training. Pick a target block. The student sees only the blocks up to it, and those are still noisy. Compare with block diffusion, which gives the earlier blocks clean.
Inference. Step through Algorithm 2 and compare with decoding one block at a time.
Loading PDF… open it directly
PDF reproduced unmodified under CC BY 4.0. Highlights and notes are mine.