dLLM Notes

Inside the Model · M02

Diffusion on discrete tokens

There is no Gaussian noise for a vocabulary.

2 min read

Every diffusion model has two halves. A fixed forward process destroys data a little at a time until nothing is left. A learned reverse process undoes it step by step. Generation runs the reverse process starting from pure noise.

For images, “a little noise” means adding a small Gaussian to every pixel. A token ID has no “a little”: token 2012 plus 0.1 is not a word. There are two ways out:

  • Diffuse in embedding space, then round back to tokens (Diffusion-LM). Rounding adds errors, and for years this line trailed the other. Nemotron-Labs-Sigma (2026) has since brought it to 8B parameters, level with a matched masked model (M12).
  • Diffuse in token space: noise means replacing tokens. This is the line LLaDA, Dream and most later dLLMs follow, and the rest of this note.

Noise as a transition matrix

Treat each token xix^i as a one-hot vector over a vocabulary of KK tokens. One forward step moves it to another token according to a K×KK \times K matrix QtQ_t, independently for every position:

q ⁣(xti∣xt−1i)=Cat ⁣(xti;  xt−1i Qt)q\!\left(x_t^i \mid x_{t-1}^i\right) = \mathrm{Cat}\!\left(x_t^i;\; x_{t-1}^i\, Q_t\right)

Row aa of QtQ_t is “where token aa goes”. Steps compose by matrix product, so any noise level is one jump from the data: q(xti∣x0i)=Cat(xti;  x0i Qˉt)q(x_t^i \mid x_0^i) = \mathrm{Cat}(x_t^i;\; x_0^i\, \bar Q_t) with Qˉt=Q1Q2⋯Qt\bar Q_t = Q_1 Q_2 \cdots Q_t. That is what makes training cheap: sample tt, corrupt x0x_0 directly, no simulation.

Everything now depends on the choice of QtQ_t. Two dominate:

UniformAbsorbing
A corrupted token becomesa random vocabulary token[MASK]
Can it change again?yes, to any tokenno, [MASK] stays [MASK]
Fully noised sequencerandom tokensall [MASK]
Uniform: random tokenAbsorbing: [MASK]t = 0ThecapitalofFranceisParis.ThecapitalofFranceisParis.t = 0.25quicklybananaofFranceisParis.ThecapitalofFranceisParis.t = 0.5quicklybananaofFranceisblue.ThecapitalofFranceisParis.t = 0.75quicklybananaofFrancethebluesangThecapitalofFranceisParis.t = 1quicklybananasevenbluethebluesangThecapitalofFranceisParis.
The same tokens are corrupted in both columns. Under uniform noise the sentence still looks like text, just wrong; tick the box to see which tokens are noise. Under absorbing noise the damage is labeled.

Why masking wins

D3PM tried both on text; absorbing noise was clearly better, and every large dLLM (LLaDA, Dream) uses it. The figure shows why:

  • The model knows where to look. A [MASK] says “fill me in”. A uniform-noise token looks like a real one, so the model must first guess which tokens are corrupt, then fix them.
  • Visible means correct. Under absorbing noise, every unmasked token is a true token. The reverse process never has to touch them, only fill the blanks.
  • The objective collapses to cross-entropy. With those two facts, the training loss becomes a weighted masked-LM loss (M03).

The price is the flip side of “visible means correct”: in the reverse process a token, once unmasked, is never revised. Errors are permanent unless the sampler adds a way to re-mask (M05).

References

  1. Austin et al. Structured Denoising Diffusion Models in Discrete State-Spaces (D3PM). NeurIPS 2021.
  2. Li et al. Diffusion-LM Improves Controllable Text Generation. NeurIPS 2022.
  3. Lou, Meng, Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (SEDD). ICML 2024.