dLLM Notes

Inside the Model · M03

Masked diffusion models

One kind of noise: [MASK]. One loss: a weighted cross-entropy.

2 min read

M02 picked the noise: replace tokens with [MASK]. This note turns that into a model and a loss.

Forward: mask each token with probability t

With the linear schedule, each token is masked independently with probability tt:

q ⁣(xti∣x0i)=(1−t) 1 ⁣[xti=x0i]+t 1 ⁣[xti=[MASK]]q\!\left(x_t^i \mid x_0^i\right) = (1 - t)\,\mathbb{1}\!\left[x_t^i = x_0^i\right] + t\,\mathbb{1}\!\left[x_t^i = \texttt{[MASK]}\right]

At t=0t = 0 the text is clean; at t=1t = 1 it is all [MASK]. In between, tt is the expected fraction masked.

Reverse: only the blanks need a model

To go from noise level tt back to a lower level s<ts < t, look at each position:

  • Visible: keep it. It is already a true token (M02).
  • Masked: with probability s/ts/t it stays masked; otherwise it is revealed as a token drawn from pθ(x0i∣xt)p_\theta(x_0^i \mid x_t).

So the only thing to learn is pθ(x0i∣xt)p_\theta(x_0^i \mid x_t): given a partly masked sequence, predict the original token at each masked position. That is a masked language model. It does not even need tt as an input, since the number of masks already says how noisy the sequence is; LLaDA has no time embedding at all.

Training in four steps

For each text x0x_0 of LL tokens in a batch:

  1. Pick a noise level. Draw t∼U(0,1)t \sim \mathcal{U}(0, 1), say t=0.56t = 0.56.
  2. Mask. Replace each token with [MASK] with probability tt. About 56% are now hidden; call the result xtx_t and the masked positions Mt\mathcal{M}_t.
  3. Predict. Run the model on xtx_t. Attention is bidirectional, so each masked position sees visible context on both sides.
  4. Score the masked positions. Sum −log⁡pθ(x0i∣xt)-\log p_\theta(x_0^i \mid x_t) over i∈Mti \in \mathcal{M}_t and multiply by 1/t1/t:
LMDLM(θ)=Et∼U(0,1), x0 ⁣[1t∑i∈Mt−log⁡pθ ⁣(x0i∣xt)]\mathcal{L}_{\text{MDLM}}(\theta) = \mathbb{E}_{t \sim \mathcal{U}(0,1),\,x_0}\!\left[\frac{1}{t}\sum_{i \in \mathcal{M}_t} -\log p_\theta\!\left(x_0^i \mid x_t\right)\right]

Then backpropagate as usual. Code typically also divides by LL for a per-token number, a constant factor.

What the loss looks like

Each bar is the loss at one masked position: short when the model is confident, long when it is guessing.

Loss at each position, −log pθ(true token | xt)

Expected loss of the whole sequence, by noise level

with the 1/t weight (the training loss)plain sum over masked tokens

Illustrative probabilities, not from a trained model. Top: one draw of masks at the chosen t. Bottom: the expected loss for this sentence at every t; the dots mark the current t. Hover the chart for values.
  • Context makes tokens cheap. At the default draw, “Paris” is masked but “capital” and “France” are visible: a short bar. “Seine” has lost most of its clues: a long one.
  • The plain sum (blue) rises for two reasons: more tokens are masked, and each is harder.
  • 1/t1/t removes the first. About tLtL tokens are masked, so dividing by tt makes every draw count as if all LL positions were scored. The weighted loss (orange) tracks difficulty only. Without it, low-tt draws would barely contribute.
Where 1/t comes from

LMDLM\mathcal{L}_{\text{MDLM}} is the continuous-time variational bound of the masked diffusion process, so

−log⁡pθ(x0)≤LMDLM.-\log p_\theta(x_0) \le \mathcal{L}_{\text{MDLM}}.

Minimizing it is maximum-likelihood training, and the bound can be compared with an AR model’s exact likelihood. For a general schedule αt\alpha_t (the probability a token is still clean), the weight is −αt′/(1−αt)-\alpha_t' / (1 - \alpha_t); the linear schedule αt=1−t\alpha_t = 1 - t gives 1/t1/t.

If you normalize, divide by LL, not by ∣Mt∣|\mathcal{M}_t|. Since ∣Mt∣≈tL|\mathcal{M}_t| \approx tL, dividing by it would cancel the 1/t1/t.

Fine-tuning on chat: never mask the prompt

Supervised fine-tuning uses (prompt, response) pairs (p,r0)(p, r_0) and changes one thing: only the response is noised. Step 2 masks each response token with probability tt and leaves the prompt intact; step 3 runs the model on [ p ; rt ][\,p\,;\,r_t\,]; step 4 scores the masked response positions:

LSFT(θ)=Et∼U(0,1), (p, r0) ⁣[1t∑i∈Mt−log⁡pθ ⁣(r0i∣p, rt)]\mathcal{L}_{\text{SFT}}(\theta) = \mathbb{E}_{t \sim \mathcal{U}(0,1),\,(p,\,r_0)}\!\left[\frac{1}{t}\sum_{i \in \mathcal{M}_t} -\log p_\theta\!\left(r_0^i \mid p,\, r_t\right)\right]

At t=1t = 1 the response is all [MASK] and the prompt is the only information left: exactly where sampling starts. M04 follows that prompt through inference.

References

  1. Sahoo et al. Simple and Effective Masked Diffusion Language Models (MDLM). NeurIPS 2024.
  2. Shi et al. Simplified and Generalized Masked Diffusion for Discrete Data (MD4). NeurIPS 2024.
  3. Ou et al. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (RADD). ICLR 2025.
  4. Nie et al. Large Language Diffusion Models (LLaDA). 2025.