Learning notes · diffusion language models
Diffusion LLMs, unmasked.
Autoregressive models write left to right, one token per forward pass. Diffusion language models start from a sequence of blanks and fill them in, several at a time, in whatever order they are most sure of. These notes follow two tracks: how the models work, and how to serve them.
Start reading →
1 / 13 unmasked
Track M · 1 / 9 published
Inside the Model
How a diffusion LLM is built, trained, and sampled: the objective, the architecture choices, and the algorithms that turn noise into text.
Foundations
- M01Why not left to right?Autoregression commits one token at a time. What if we did not?Sep 2026
- M02Diffusion on discrete tokensThere is no Gaussian noise for a vocabulary.planned
- M03Masked diffusion modelsOne kind of noise: [MASK]. One loss: a weighted cross-entropy.planned
- M04Sampling and remaskingWhich blanks do we fill, and when do we change our mind?planned
Scaling up
- M05LLaDADoes masked diffusion scale like autoregression?planned
- M06Starting from an AR checkpointDream and DiffuLLaMA: reuse the weights, change the objective.planned
- M07Block diffusionAutoregressive across blocks, diffusion within them.planned
Steering
- M08Constrained decodingGrammars, when the order is not left to right.planned
- M09Reinforcement learning for dLLMsPolicy gradients without a tractable likelihood.planned
Track P · 0 / 4 published
Into Production
What it takes to serve a diffusion LLM fast and cheaply: caching, parallel decoding, batching, and the shape of an inference engine.
Making one request fast
- P01Caching without causalityBidirectional attention breaks the KV cache. Mostly.planned
- P02Parallel decodingHow many tokens can we commit in one step?planned
Serving many requests
- P03Batching denoising stepsEvery request is at a different noise level. Can they share a step?planned
- P04A serving engine for dLLMsScheduler, cache manager, sampler: what changes from an AR engine.planned