Inside the Model · M03
Masked diffusion models
One kind of noise: [MASK]. One loss: a weighted cross-entropy.
M02 picked the noise: replace tokens with [MASK]. This note turns that into a model and a loss.
Forward: mask each token with probability t
With the linear schedule, each token is masked independently with probability :
At the text is clean; at it is all [MASK]. In between, is the expected fraction masked.
Reverse: only the blanks need a model
To go from noise level back to a lower level , look at each position:
- Visible: keep it. It is already a true token (M02).
- Masked: with probability it stays masked; otherwise it is revealed as a token drawn from .
So the only thing to learn is : given a partly masked sequence, predict the original token at each masked position. That is a masked language model. It does not even need as an input, since the number of masks already says how noisy the sequence is; LLaDA has no time embedding at all.
Training in four steps
For each text of tokens in a batch:
- Pick a noise level. Draw , say .
- Mask. Replace each token with
[MASK]with probability . About 56% are now hidden; call the result and the masked positions . - Predict. Run the model on . Attention is bidirectional, so each masked position sees visible context on both sides.
- Score the masked positions. Sum over and multiply by :
Then backpropagate as usual. Code typically also divides by for a per-token number, a constant factor.
What the loss looks like
Each bar is the loss at one masked position: short when the model is confident, long when it is guessing.
Loss at each position, −log pθ(true token | xt)
Expected loss of the whole sequence, by noise level
with the 1/t weight (the training loss)plain sum over masked tokens
- Context makes tokens cheap. At the default draw, “Paris” is masked but “capital” and “France” are visible: a short bar. “Seine” has lost most of its clues: a long one.
- The plain sum (blue) rises for two reasons: more tokens are masked, and each is harder.
- removes the first. About tokens are masked, so dividing by makes every draw count as if all positions were scored. The weighted loss (orange) tracks difficulty only. Without it, low- draws would barely contribute.
Where 1/t comes from
is the continuous-time variational bound of the masked diffusion process, so
Minimizing it is maximum-likelihood training, and the bound can be compared with an AR model’s exact likelihood. For a general schedule (the probability a token is still clean), the weight is ; the linear schedule gives .
If you normalize, divide by , not by . Since , dividing by it would cancel the .
Fine-tuning on chat: never mask the prompt
Supervised fine-tuning uses (prompt, response) pairs and changes one thing: only the response is noised. Step 2 masks each response token with probability and leaves the prompt intact; step 3 runs the model on ; step 4 scores the masked response positions:
At the response is all [MASK] and the prompt is the only information left: exactly where sampling starts. M04 follows that prompt through inference.
References
- Sahoo et al. Simple and Effective Masked Diffusion Language Models (MDLM). NeurIPS 2024.
- Shi et al. Simplified and Generalized Masked Diffusion for Discrete Data (MD4). NeurIPS 2024.
- Ou et al. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (RADD). ICLR 2025.
- Nie et al. Large Language Diffusion Models (LLaDA). 2025.