Learning notes · diffusion language models
Diffusion LLMs, unmasked.
prompt
<user>WhatisthecapitalofTaiwan?<assistant>
answer
ThecapitalofTaiwanisTaipei.<eos><eos><eos>
18 / 30 unmasked
Track M · 10 / 12 published
Inside the Model
How a diffusion LLM is built, trained, and sampled: the objective, the architecture choices, and the algorithms that turn noise into text.
Foundations
- M01Why not left to right?Autoregression commits one token at a time. What if we did not?Sep 2026
- M02Diffusion on discrete tokensThere is no Gaussian noise for a vocabulary.Oct 2026
- M03Masked diffusion modelsOne kind of noise: [MASK]. One loss: a weighted cross-entropy.Oct 2026
- M04Where the prompt goesOnly the answer starts as noise.Oct 2026
- M05Sampling and remaskingWhich blanks do we fill, and when do we change our mind?Oct 2026
Scaling up
- M06LLaDA, Dream, and what came nextDoes masked diffusion scale like autoregression? Two open answers, and the versions that followed.Nov 2026
- M07Starting from an AR checkpointReuse the AR weights; change the mask and the objective.Oct 2026
- M08Block diffusionAutoregressive across blocks, diffusion within them.Nov 2026
- M09Mixture-of-Experts dLLMsSparse experts, dense blocks: every token in the block wakes up its own experts.Oct 2026
Steering
- M10Constrained decodingGrammars, when the order is not left to right.planned
- M11Reinforcement learning for dLLMsPolicy gradients without a tractable likelihood.planned
Track P · 6 / 11 published
Into Production
What it takes to serve a diffusion LLM fast and cheaply: caching, parallel decoding, batching, and the shape of an inference engine.
Caching mechanisms
- P01Caching without causalityBidirectional attention breaks the KV cache. Mostly.Oct 2026
- P02Keeping fewer tokensMost keys get almost no attention. Which ones can each head drop?Nov 2026
- P03An exact cache againMake attention causal between blocks, and the KV cache stops being an approximation.Nov 2026
- P04Trimming the masked suffixHundreds of trailing masks, and the model barely looks past the first few.Nov 2026
Decoding strategies
- P05Parallel decodingHow many tokens can we commit in one step?Nov 2026
- P06Draft, verify, remaskCommit aggressively, then let bidirectional context take the wrong ones back.planned
- P07Reading the denoising trajectoryThe answer often settles long before the last step. Stop there, or let the steps vote.planned
Serving many requests
Serving engines
- P09dLLMs in vLLM and SGLangScheduler, KV cache, sampler: what carries over from an AR engine, what bends, and what is new.planned
- P10Engines built for dLLMsdInfer, dLLM-Serve, Flash-dLLM: what they do that general engines do not, yet.planned
- P11Attention masks in the kernelCausal for some requests, bidirectional for others, in the same batch.planned
Track S · 2 / 7 published
Speculative Drafting
Serving AR models faster with ideas from diffusion: a cheap drafter guesses many tokens, the target checks them all in one pass, and the output stays exactly the same.
Drafters
- S02AR drafters: Medusa to EAGLE-3Draft from the target's own features, one token per pass.planned
- S03Diffusion drafters: DFlash 1 and 2A whole block in one pass. Then make the block hold together.Nov 2026
- S04DSparkBetween left to right and all at once, and verify only what is likely to pass.planned
- S05Draft treesCheck several guesses at once, in one pass, with a tree mask.planned
- S06Drafting for yourselfOne model, two heads: diffusion drafts, AR checks.planned
In production
- S07Speculation under loadAt batch 1 verification is free. At batch 32 it is not.planned
Reading · 13 papers
Paper notes
The papers behind the notes, as the original PDFs with my highlights and margin notes.
- 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory DistillationICMLLearn the decoding order from the teacher, decode several blocks at once, and score speed and accuracy as one curve.Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, Hao Zhang
- 2026DFlash: Block Diffusion for Flash Speculative DecodingICMLA five-layer block diffusion drafter, fed with the target model's hidden features, guesses 16 tokens in one pass.Jian Chen, Yesheng Liang, Zhijian Liu
- 2026Large Language Continuous Diffusion Models (Nemotron-Labs-Sigma)arXivGaussian noise on 16-dimensional token embeddings, at 3B and 8B, against a masked model trained the same way.Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Mohammad Mahdi Kamani, Yonggan Fu, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Jukić, Arash Vahdat, Morteza Mardani
- 2026Multi-Block Diffusion Language Models (MBD-LMs)arXivDecode two or three blocks at once, and train for what that actually looks like.Yijie Jin, Jiajun Xu, Yuxuan Liu, Chenkai Xu, Yi Tu, Jiajun Li, Dandan Tu, Xiaohui Yan, Kai Yu, Pengfei Liu, Zhijie Deng
- 2025DPad: Efficient Diffusion Language Models with Suffix DropoutarXivMost of the masked suffix is never needed. Keep a window of nearby masks, thin it with a Gaussian, and drop the rest before the forward pass.Xinhua Chen, Sitao Huang, Cong Guo, Chiyue Wei, Yintao He, Jianyi Zhang, Hai "Helen" Li, Yiran Chen
- 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing (D2F)arXivDistill a full-attention dLLM into a block-causal one that can start the next block before the last one is done.Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, Zhijie Deng
- 2025Attention Is All You Need for KV Cache in Diffusion LLMs (Elastic-Cache)arXivRefresh the cache only when the most-attended token moves, and only from that layer down. Masks far to the right stay cached.Quan Nguyen-Tri, Mukul Ranjan, Zhiqiang Shen
- 2025Fast-dLLM v2: Efficient Block-Diffusion LLMarXivTurn Qwen2.5 into a block diffusion model with ~1B tokens of fine-tuning, then decode with a block-level cache plus a sub-block DualCache.Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, Enze Xie
- 2025Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM ServingarXivA serving system for dLLMs: cap the logit spike, schedule Refresh and Reuse phases separately, store per-head sparse KV densely.Jiakun Fan, Yanglin Zhang, Xiangchen Li, Dimitrios S. Nikolopoulos
- 2026TEAM: Temporal–Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model AccelerationarXivMoE dLLMs load far more experts per step than they accept tokens. Cache decoded tokens, fence in cold masks, speculate on hot ones.Linye Wei, Zixiang Luo, Pingzhi Tang, Meng Li
- 2025dKV-Cache: The Cache for Diffusion Language ModelsNeurIPSCache a token only after it is decoded, and one step late. No blocks, no retraining.Xinyin Ma, Runpeng Yu, Gongfan Fang, Xinchao Wang
- 2026dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive CachingICMLRefresh the prompt rarely, the response often, and only the response tokens that actually changed.Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyan Wei, Shaobo Wang, Yichen Zhu, Linfeng Zhang
- 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel DecodingarXivAn approximate KV cache for bidirectional attention, plus confidence-thresholded parallel decoding. No retraining.Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, Enze Xie