dLLM Notes

Paper notes · ICML 2026

d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation

Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, Hao Zhang

d3LLM speeds up LLaDA and Dream, both full-attention models, by short distillation (65M tokens). It argues that a dLLM always trades accuracy for parallelism, and that most methods improve only one side. Three parts:

PromptWhat do you think of dLLM?

    Figure 2 of the paper, step by step. The teacher decodes the current block one token per step (its third token is wrong). Only its order is kept. The mask ratio t then masks the positions it filled last, and the student learns to predict them from the ground truth. Drag t, or switch to random masking to compare.

    Result on LLaDA, GSM8K: 9.11 tokens per forward at 73.1% accuracy, against 1.00 at 72.6% for vanilla LLaDA (Table 1). It comes from the same circle as D2F and MBD-LMs (notes), and MBD-LMs adopt its AUP. In my notes: AUP in P04, the distillation in P07, the SGLang port in P09.

    The PDF's license does not allow reposting it here; page and line numbers refer to the version linked above.