dLLM Notes

Inside the Model · M05

Sampling and remasking

Which blanks do we fill, and when do we change our mind?

3 min read

M03 trained pθ(x0i∣xt)p_\theta(x_0^i \mid x_t); M04 set up the canvas. Sampling is the loop that connects them, and almost every quality and speed trade-off in a dLLM lives here.

The loop

Start from the prompt and NN masks. Pick a number of steps TT, so about k=N/Tk = N / T tokens are committed per step. Then repeat:

  1. Predict. One forward pass gives a distribution pθ(⋅∣xt)p_\theta(\cdot \mid x_t) at every masked position.
  2. Choose. Pick which masked positions to commit this step.
  3. Commit. Fill those with a token sampled (or the argmax) from their distribution. Every other position stays [MASK], even if the model already had a guess for it.

After TT steps nothing is masked. TT is the main speed knob: T=NT = N is one token per pass, like AR; smaller TT is faster and riskier.

Which positions to commit

The reverse process from M03 says: reveal each masked position with equal probability, so choose at random. It is correct in the limit of many steps, but in practice it is not the best rule. Most samplers rank positions by how sure the model is:

RuleCommit the positions withUsed by
Randoma random subsetthe plain reverse process
Confidencethe highest top-1 probabilityLLaDA (“low-confidence remasking”)
Marginthe largest gap between top-1 and top-2Dream (option)
Entropythe lowest entropyDream (option)

Committing confident tokens first means easy tokens (punctuation, the obvious next word, trailing <eos>) are fixed early and then serve as context for the hard ones. LLaDA’s name for the rule describes step 3: predictions that did not make the cut are thrown away and those positions are masked again.

How many per step

Tokens committed in the same step are sampled independently, each from its own marginal. The model below is torn between “New York” and “San Francisco”:

Both in one step1 pass
[MASK][MASK]

New York
San Francisco
New Francisco
San York
One per step2 passes
[MASK][MASK]

New York
San Francisco
New Francisco
San York

A toy model whose joint only allows the two real cities. Sample a few times, then drag the slider.

The risk depends on confidence, so a fixed kk is a blunt tool. A confidence threshold adapts it: commit every position whose top probability exceeds τ\tau (at least one per step). Sure stretches go fast, torn ones slow down. Fast-dLLM proves when this matches sequential decoding (my paper notes); P05 goes deeper.

Semi-autoregressive blocks

Pure confidence ordering over a long canvas has a failure mode. The tail of an oversized canvas is <eos> padding, which the model is very sure of, so it gets committed first, and an instruct model can end up writing answers that are too short. LLaDA’s fix is to split the canvas into blocks of BB tokens and sample them left to right, running the diffusion loop within the current block only.

Block size is a dial between the two worlds: B=NB = N is plain diffusion; B=1B = 1 is autoregression. M08 (block diffusion) builds the block structure into training as well.

Remasking: changing its mind

Under absorbing noise a committed token is final (M02). An early mistake stays, and later tokens are generated around it. Remasking samplers relax this: at each step they may also send some already-committed tokens back to [MASK], typically low-confidence ones, so they get predicted again with more context. ReMDM does this in a principled way and shows quality improving as more steps are spent, a form of inference-time scaling.

References

  1. Nie et al. Large Language Diffusion Models (LLaDA). 2025.
  2. Ye et al. Dream 7B: Diffusion Large Language Models. 2025.
  3. Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. 2025.
  4. Wang et al. Remasking Discrete Diffusion Models with Inference-Time Scaling (ReMDM). 2025.