dLLM Notes

Learning notes · diffusion language models

Diffusion LLMs, unmasked.

prompt
<user>WhatisthecapitalofTaiwan?<assistant>
answer
ThecapitalofTaiwanisTaipei.<eos><eos><eos>

Track M · 10 / 12 published

Inside the Model

How a diffusion LLM is built, trained, and sampled: the objective, the architecture choices, and the algorithms that turn noise into text.

Steering

  1. M10Constrained decodingGrammars, when the order is not left to right.planned
  2. M11Reinforcement learning for dLLMsPolicy gradients without a tractable likelihood.planned

Track P · 6 / 11 published

Into Production

What it takes to serve a diffusion LLM fast and cheaply: caching, parallel decoding, batching, and the shape of an inference engine.

Decoding strategies

  1. P05Parallel decodingHow many tokens can we commit in one step?Nov 2026
  2. P06Draft, verify, remaskCommit aggressively, then let bidirectional context take the wrong ones back.planned
  3. P07Reading the denoising trajectoryThe answer often settles long before the last step. Stop there, or let the steps vote.planned

Serving engines

  1. P09dLLMs in vLLM and SGLangScheduler, KV cache, sampler: what carries over from an AR engine, what bends, and what is new.planned
  2. P10Engines built for dLLMsdInfer, dLLM-Serve, Flash-dLLM: what they do that general engines do not, yet.planned
  3. P11Attention masks in the kernelCausal for some requests, bidirectional for others, in the same batch.planned

Track S · 2 / 7 published

Speculative Drafting

Serving AR models faster with ideas from diffusion: a cheap drafter guesses many tokens, the target checks them all in one pass, and the output stays exactly the same.

Drafters

  1. S02AR drafters: Medusa to EAGLE-3Draft from the target's own features, one token per pass.planned
  2. S03Diffusion drafters: DFlash 1 and 2A whole block in one pass. Then make the block hold together.Nov 2026
  3. S04DSparkBetween left to right and all at once, and verify only what is likely to pass.planned
  4. S05Draft treesCheck several guesses at once, in one pass, with a tree mask.planned
  5. S06Drafting for yourselfOne model, two heads: diffusion drafts, AR checks.planned

In production

  1. S07Speculation under loadAt batch 1 verification is free. At batch 32 it is not.planned

Reading · 13 papers

Paper notes

The papers behind the notes, as the original PDFs with my highlights and margin notes.

  1. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory DistillationICMLLearn the decoding order from the teacher, decode several blocks at once, and score speed and accuracy as one curve.Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, Hao Zhang
  2. 2026DFlash: Block Diffusion for Flash Speculative DecodingICMLA five-layer block diffusion drafter, fed with the target model's hidden features, guesses 16 tokens in one pass.Jian Chen, Yesheng Liang, Zhijian Liu
  3. 2026Large Language Continuous Diffusion Models (Nemotron-Labs-Sigma)arXivGaussian noise on 16-dimensional token embeddings, at 3B and 8B, against a masked model trained the same way.Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Mohammad Mahdi Kamani, Yonggan Fu, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Jukić, Arash Vahdat, Morteza Mardani
  4. 2026Multi-Block Diffusion Language Models (MBD-LMs)arXivDecode two or three blocks at once, and train for what that actually looks like.Yijie Jin, Jiajun Xu, Yuxuan Liu, Chenkai Xu, Yi Tu, Jiajun Li, Dandan Tu, Xiaohui Yan, Kai Yu, Pengfei Liu, Zhijie Deng
  5. 2025DPad: Efficient Diffusion Language Models with Suffix DropoutarXivMost of the masked suffix is never needed. Keep a window of nearby masks, thin it with a Gaussian, and drop the rest before the forward pass.Xinhua Chen, Sitao Huang, Cong Guo, Chiyue Wei, Yintao He, Jianyi Zhang, Hai "Helen" Li, Yiran Chen
  6. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing (D2F)arXivDistill a full-attention dLLM into a block-causal one that can start the next block before the last one is done.Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, Zhijie Deng
  7. 2025Attention Is All You Need for KV Cache in Diffusion LLMs (Elastic-Cache)arXivRefresh the cache only when the most-attended token moves, and only from that layer down. Masks far to the right stay cached.Quan Nguyen-Tri, Mukul Ranjan, Zhiqiang Shen
  8. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMarXivTurn Qwen2.5 into a block diffusion model with ~1B tokens of fine-tuning, then decode with a block-level cache plus a sub-block DualCache.Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, Enze Xie
  9. 2025Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM ServingarXivA serving system for dLLMs: cap the logit spike, schedule Refresh and Reuse phases separately, store per-head sparse KV densely.Jiakun Fan, Yanglin Zhang, Xiangchen Li, Dimitrios S. Nikolopoulos
  10. 2026TEAM: Temporal–Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model AccelerationarXivMoE dLLMs load far more experts per step than they accept tokens. Cache decoded tokens, fence in cold masks, speculate on hot ones.Linye Wei, Zixiang Luo, Pingzhi Tang, Meng Li
  11. 2025dKV-Cache: The Cache for Diffusion Language ModelsNeurIPSCache a token only after it is decoded, and one step late. No blocks, no retraining.Xinyin Ma, Runpeng Yu, Gongfan Fang, Xinchao Wang
  12. 2026dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive CachingICMLRefresh the prompt rarely, the response often, and only the response tokens that actually changed.Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyan Wei, Shaobo Wang, Yichen Zhu, Linfeng Zhang
  13. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel DecodingarXivAn approximate KV cache for bidirectional attention, plus confidence-thresholded parallel decoding. No retraining.Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, Enze Xie