dLLM Notes

Inside the Model · M08

Block diffusion

Autoregressive across blocks, diffusion within them.

5 min read

LLaDA can already decode in blocks: split the canvas into blocks and run the diffusion loop on one block at a time, left to right (M05). Block diffusion does more: it builds the blocks into the model. Attention is bidirectional inside a block and causal across blocks, and the model is trained that way.

Same order, different mask

Both models below decode the same answer in the same order. Step through and watch the prompt and block 1 once block 2 starts.

Full attention, decoded block by blockLLaDA, Dream (semi-AR sampling)
promptblock 1block 2block 3

row attends to column

Block diffusionBD3-LM, SDAR, Fast-dLLM v2, LLaDA 2.0
promptblock 1block 2block 3

row attends to column

Top: LLaDA-style full attention with block-by-block sampling. Finished blocks attend to the block being decoded, so their K/V go stale every step, and the masks on the right are computed long before they are decoded. Bottom: block diffusion. Earlier blocks cannot attend to later ones, so once a block is finished its K/V are final. Each step runs only the current block, and the sequence grows until eos.

Three differences follow from the mask:

  1. Exact KV cache. A finished block’s K/V depend only on itself and earlier blocks, which never change again. So the cache is exact, as in AR. With full attention, any cache is an approximation that needs refreshing (P01).
  2. Less work per step. Only the current block runs. Full attention also runs every mask to the right, every step.
  3. Variable length. Full attention needs the canvas length before decoding starts (M04). Block diffusion appends blocks until it emits eos, like AR. BD3-LM samples up to ~10K tokens with a 1024-token training context; SEDD, a full-attention model, is capped at 1024.

The model

Split the sequence xx of length LL into BB blocks of L′L' tokens. Factorize over blocks, as AR factorizes over tokens:

log⁡pθ(x)=∑b=1Blog⁡pθ(xb∣x<b)\log p_\theta(x) = \sum_{b=1}^{B} \log p_\theta\left(x^b \mid x^{<b}\right)

Each factor is a masked diffusion model over one block, conditioned on the clean blocks before it. Its loss is the masked diffusion loss of M03, restricted to block bb.

Block size is a dial between the two families:

  • L′=1L' = 1: one token per block, which is AR.
  • L′=LL' = L: one block, which is plain masked diffusion (MDLM, LLaDA).

Between them, smaller blocks are closer to AR in quality but give less to decode in parallel. BD3-LM on OpenWebText, 110M parameters (perplexities of diffusion models are upper bounds):

ModelTest PPL ↓Gen. PPL ↓ (1024 tokens)
AR17.5414.1
BD3-LM, L′=4L' = 4≤ 20.7325.7
BD3-LM, L′=8L' = 8≤ 21.6830.4
BD3-LM, L′=16L' = 16≤ 22.2733.4
MDLM (L′=LL' = L)≤ 22.9846.8

Training: two copies in one pass

A block is decoded while the blocks before it are clean. So training noises each block with its own level tbt_b, and the noisy block bb must see the clean versions of blocks <b< b. A single noisy copy cannot give both.

BD3-LM’s fix is to concatenate the noisy and clean sequences, [xt,x0][x_t, x_0] with length 2L2L, and train all blocks in one forward pass with this mask:

(a) Masked diffusion

input xt, one noise level t

The▢▢on▢matThe▢▢on▢mat

(b) Block diffusion

input [xt, x0], one noise level tb per block of 2

▢cat▢▢a▢Thecatsatonamat▢cat▢▢a▢Thecatsatonamat

xt (noisy)x0 (clean)

rows: the token being computed · columns: tokens it can attend to

Left: masked diffusion trains on one noisy copy with one noise level and full attention. Right: block diffusion trains on [x_t, x_0]. A noisy block attends to itself both ways (orange) and to the clean earlier blocks (green). The clean copy is block-causal (blue) and supplies those clean K/V, like teacher forcing. Nothing attends from x_0 to x_t. The loss is taken on the masked tokens of x_t only.

The price is twice the tokens per forward pass, and about three quarters of the 2L×2L2L \times 2L mask is empty. Fast-dLLM v2 uses the same scheme and runs it with FlexAttention, which skips the empty tiles (my notes).

A surprise at L′=1L' = 1

At L′=1L' = 1 the block diffusion loss equals the AR loss in expectation, yet BD3-LM first trained to perplexity ≤ 25.56 against AR’s 22.88 on LM1B. The cause is variance, not the model. With t∼U(0,1)t \sim U(0, 1), half the tokens are masked on average, so each batch gives gradients from half as many tokens as AR’s. Masking every token at L′=1L' = 1 makes the loss exactly AR’s and closes the gap (22.88).

For larger blocks, BD3-LM samples the mask rate from a clipped range U[β,ω]U[\beta, \omega] instead of U[0,1]U[0, 1]: almost-clean and fully masked blocks teach little and add noise. The best range depends on L′L' and is tuned during training.

What it costs

  • Context on the right. A token sees the rest of its block, but never later blocks. Full attention can revise early text in light of later text; block diffusion cannot.
  • Blocks run in sequence. Parallelism exists only inside a block. With small blocks, decoding is close to AR in the number of forward passes. The cause is in training, not in the mask (see below).
  • The current block. Inside it, attention is still bidirectional, so a cache within the block is again approximate. Fast-dLLM v2 adds DualCache at the sub-block level for this.

Teacher forcing vs. diffusion forcing

Why must block b+1b+1 wait for block bb? Because of how BD3-LM trains. The noisy block always conditions on clean earlier blocks, the x0x_0 half of [xt,x0][x_t, x_0]. This is teacher forcing: the model never sees an unfinished prefix, so at inference the prefix must be finished first.

D2F (my notes) keeps the block-causal mask and changes the training input. Every block is noisy, with mask ratios rising from left to right (t1<t2<…t_1 < t_2 < \dots). So block bb learns to denoise from earlier blocks that are still partly masked. This is diffusion forcing, a name borrowed from video models, where each frame gets its own noise level. At inference, D2F appends a new block once the last one is only partly decoded and decodes several blocks in one forward pass. Finished blocks are still cached exactly.

Earlier blocks during trainingAt inference, block b+1b+1 starts
Block diffusion (BD3-LM, Fast-dLLM v2)cleanwhen block bb is finished
D2Fnoisy, less masked than block bbwhen block bb is partly decoded
MultiTF (MBD-LMs)clean before a group of 2–4 blocks; inside it, noisy with large gapswhen block bb is partly decoded

MultiTF (my notes) sits between the two. At inference, D2F-style decoding usually has only about two active blocks, and their mask ratios are far apart: one nearly done, the next mostly masked. D2F trains on a long run of blocks whose mask ratios rise smoothly, which rarely looks like that. MultiTF post-trains a block diffusion model (LLaDA2, SDAR) on a clean prefix plus a small group of noisy blocks, with random and often large gaps between their mask ratios. With D2F’s smooth schedule instead, accuracy drops by about 5 points.

D2F gets its block-causal model by distillation from LLaDA or Dream, and not from an AR checkpoint. That is a separate choice from the forcing.

Who uses it

The larger recent dLLMs are block diffusion models converted from AR checkpoints (M07): SDAR, Fast-dLLM v2 and LLaDA 2.0. The caching side, where finished blocks form an exact cache, is P03.

References

  1. Arriola et al. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LM). ICLR 2025. arXiv
  2. Sahoo et al. Simple and Effective Masked Diffusion Language Models (MDLM). NeurIPS 2024.
  3. Wu et al. Fast-dLLM v2: Efficient Block-Diffusion LLM. ICLR 2026.
  4. Cheng et al. SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation. 2025.
  5. Bie et al. LLaDA2.0: Scaling Up Diffusion Language Models to 100B. 2025.
  6. Wang et al. Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing (D2F). 2025.
  7. Jin et al. Multi-Block Diffusion Language Models. 2026. arXiv
  8. Chen et al. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. NeurIPS 2024.