dLLM Notes

Paper notes · arXiv 2025

DPad: Efficient Diffusion Language Models with Suffix Dropout

Xinhua Chen, Sitao Huang, Cong Guo, Chiyue Wei, Yintao He, Jianyi Zhang, Hai "Helen" Li, Yiran Chen

A training-free method for full-attention dLLMs (LLaDA, Dream). Every step of vanilla decoding runs the whole masked suffix, though only the current block is decoded. DPad cuts the suffix down before the forward pass:

The kept set is drawn once per block, the RoPE positions of kept tokens stay the original ones, and DPad stacks with parallel decoding and the prefix cache. It also stops at the first block that contains eos.

One block of 8 followed by 96 suffix masks. The curve is the keep probability P(d) of Eq. 10; each draw keeps a different subset. Move W, k and a and watch the density: k sets how quickly P falls inside the window, a the overall level. Toy: one coin flip per position; the paper samples with rejection sampling.

Loading PDF… open it directly

PDF reproduced unmodified under CC BY 4.0. Highlights and notes are mine.