Paper notes · ICML 2026
d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation
d3LLM speeds up LLaDA and Dream, both full-attention models, by short distillation (65M tokens). It argues that a dLLM always trades accuracy for parallelism, and that most methods improve only one side. Three parts:
- A metric (§2, AUP): score a method by its whole accuracy–parallelism curve, not one point.
- Training (§3.1, pseudo-trajectory distillation): teach the student to skip steps. The teacher decodes one token per step. The student is shown a state from early in the teacher’s run and trained to predict, at once, the tokens the teacher only filled several steps later. In those states it becomes confident about more positions, so more pass the threshold per step, more tokens are committed per forward, and the teacher’s steps shrink to fewer.
- Inference (§3.2): multi-block decoding as in D2F, with a periodic KV-cache refresh.
PromptWhat do you think of dLLM?
Result on LLaDA, GSM8K: 9.11 tokens per forward at 73.1% accuracy, against 1.00 at 72.6% for vanilla LLaDA (Table 1). It comes from the same circle as D2F and MBD-LMs (notes), and MBD-LMs adopt its AUP. In my notes: AUP in P04, the distillation in P07, the SGLang port in P09.
The PDF's license does not allow reposting it here; page and line numbers refer to the version linked above.