dLLM Notes

Inside the Model · M01

Why not left to right?

Autoregression commits one token at a time. What if we did not?

1 min read

Every mainstream LLM writes the same way: predict one more token, append it, repeat. That bakes in two assumptions: text is produced in order, and one token per forward pass. Diffusion language models (dLLMs) drop both.

Two ways to write a sentence

An autoregressive (AR) model factorizes a sequence x=(x1,…,xL)x = (x^1, \dots, x^L) left to right:

pθ(x)=∏i=1Lpθ ⁣(xi∣x<i)p_\theta(x) = \prod_{i=1}^{L} p_\theta\!\left(x^i \mid x^{<i}\right)

So LL tokens take LL forward passes, and an emitted token is never revisited.

A masked diffusion model starts from LL positions that are all [MASK]. Each step, the network reads the whole sequence (attention is bidirectional), predicts every masked position, and keeps the predictions it is most confident about. A few steps later nothing is masked.

Autoregressive
Themodelfillsineveryblankatonce,thenkeepstheonesitissureabout.
Masked diffusion
Themodelfillsineveryblankatonce,thenkeepstheonesitissureabout.
Both rows generate the same 18 tokens. The AR model needs 18 forward passes; the diffusion model needs ⌈18 / k⌉. Drag the slider to change k, the number of tokens committed per step.

Two things change: order becomes a choice (the end of a sentence can be fixed before the middle), and the number of steps is a knob, not LL.

What it costs

  • Parallel tokens are guessed independently. If the model is split between “New York” and “San Francisco”, committing both words in one step can give “New Francisco”. Fewer tokens per step avoids this, at the price of more steps (M05).
  • No free KV cache. With bidirectional attention every position can change every other position’s representation, so the AR cache does not apply directly (M04, P01).
  • Fixed canvas. Sampling starts from a fixed number of masked positions, so the length has to be chosen up front (M04).

References

  1. Austin et al. Structured Denoising Diffusion Models in Discrete State-Spaces (D3PM). NeurIPS 2021.
  2. Sahoo et al. Simple and Effective Masked Diffusion Language Models (MDLM). NeurIPS 2024.
  3. Nie et al. Large Language Diffusion Models (LLaDA). 2025.