Inside the Model · M02
Diffusion on discrete tokens
There is no Gaussian noise for a vocabulary.
Every diffusion model has two halves. A fixed forward process destroys data a little at a time until nothing is left. A learned reverse process undoes it step by step. Generation runs the reverse process starting from pure noise.
For images, “a little noise” means adding a small Gaussian to every pixel. A token ID has no “a little”: token 2012 plus 0.1 is not a word. There are two ways out:
- Diffuse in embedding space, then round back to tokens (Diffusion-LM). Rounding adds errors, and for years this line trailed the other. Nemotron-Labs-Sigma (2026) has since brought it to 8B parameters, level with a matched masked model (M12).
- Diffuse in token space: noise means replacing tokens. This is the line LLaDA, Dream and most later dLLMs follow, and the rest of this note.
Noise as a transition matrix
Treat each token as a one-hot vector over a vocabulary of tokens. One forward step moves it to another token according to a matrix , independently for every position:
Row of is “where token goes”. Steps compose by matrix product, so any noise level is one jump from the data: with . That is what makes training cheap: sample , corrupt directly, no simulation.
Everything now depends on the choice of . Two dominate:
| Uniform | Absorbing | |
|---|---|---|
| A corrupted token becomes | a random vocabulary token | [MASK] |
| Can it change again? | yes, to any token | no, [MASK] stays [MASK] |
| Fully noised sequence | random tokens | all [MASK] |
Why masking wins
D3PM tried both on text; absorbing noise was clearly better, and every large dLLM (LLaDA, Dream) uses it. The figure shows why:
- The model knows where to look. A
[MASK]says “fill me in”. A uniform-noise token looks like a real one, so the model must first guess which tokens are corrupt, then fix them. - Visible means correct. Under absorbing noise, every unmasked token is a true token. The reverse process never has to touch them, only fill the blanks.
- The objective collapses to cross-entropy. With those two facts, the training loss becomes a weighted masked-LM loss (M03).
The price is the flip side of “visible means correct”: in the reverse process a token, once unmasked, is never revised. Errors are permanent unless the sampler adds a way to re-mask (M05).
References
- Austin et al. Structured Denoising Diffusion Models in Discrete State-Spaces (D3PM). NeurIPS 2021.
- Li et al. Diffusion-LM Improves Controllable Text Generation. NeurIPS 2022.
- Lou, Meng, Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (SEDD). ICML 2024.