Inside the Model · M08
Block diffusion
Autoregressive across blocks, diffusion within them.
LLaDA can already decode in blocks: split the canvas into blocks and run the diffusion loop on one block at a time, left to right (M05). Block diffusion does more: it builds the blocks into the model. Attention is bidirectional inside a block and causal across blocks, and the model is trained that way.
Same order, different mask
Both models below decode the same answer in the same order. Step through and watch the prompt and block 1 once block 2 starts.
row attends to column
row attends to column
Three differences follow from the mask:
- Exact KV cache. A finished block’s K/V depend only on itself and earlier blocks, which never change again. So the cache is exact, as in AR. With full attention, any cache is an approximation that needs refreshing (P01).
- Less work per step. Only the current block runs. Full attention also runs every mask to the right, every step.
- Variable length. Full attention needs the canvas length before decoding starts (M04). Block diffusion appends blocks until it emits eos, like AR. BD3-LM samples up to ~10K tokens with a 1024-token training context; SEDD, a full-attention model, is capped at 1024.
The model
Split the sequence of length into blocks of tokens. Factorize over blocks, as AR factorizes over tokens:
Each factor is a masked diffusion model over one block, conditioned on the clean blocks before it. Its loss is the masked diffusion loss of M03, restricted to block .
Block size is a dial between the two families:
- : one token per block, which is AR.
- : one block, which is plain masked diffusion (MDLM, LLaDA).
Between them, smaller blocks are closer to AR in quality but give less to decode in parallel. BD3-LM on OpenWebText, 110M parameters (perplexities of diffusion models are upper bounds):
| Model | Test PPL ↓ | Gen. PPL ↓ (1024 tokens) |
|---|---|---|
| AR | 17.54 | 14.1 |
| BD3-LM, | ≤ 20.73 | 25.7 |
| BD3-LM, | ≤ 21.68 | 30.4 |
| BD3-LM, | ≤ 22.27 | 33.4 |
| MDLM () | ≤ 22.98 | 46.8 |
Training: two copies in one pass
A block is decoded while the blocks before it are clean. So training noises each block with its own level , and the noisy block must see the clean versions of blocks . A single noisy copy cannot give both.
BD3-LM’s fix is to concatenate the noisy and clean sequences, with length , and train all blocks in one forward pass with this mask:
(a) Masked diffusion
input xt, one noise level t
(b) Block diffusion
input [xt, x0], one noise level tb per block of 2
xt (noisy)x0 (clean)
rows: the token being computed · columns: tokens it can attend to
The price is twice the tokens per forward pass, and about three quarters of the mask is empty. Fast-dLLM v2 uses the same scheme and runs it with FlexAttention, which skips the empty tiles (my notes).
A surprise at
At the block diffusion loss equals the AR loss in expectation, yet BD3-LM first trained to perplexity ≤ 25.56 against AR’s 22.88 on LM1B. The cause is variance, not the model. With , half the tokens are masked on average, so each batch gives gradients from half as many tokens as AR’s. Masking every token at makes the loss exactly AR’s and closes the gap (22.88).
For larger blocks, BD3-LM samples the mask rate from a clipped range instead of : almost-clean and fully masked blocks teach little and add noise. The best range depends on and is tuned during training.
What it costs
- Context on the right. A token sees the rest of its block, but never later blocks. Full attention can revise early text in light of later text; block diffusion cannot.
- Blocks run in sequence. Parallelism exists only inside a block. With small blocks, decoding is close to AR in the number of forward passes. The cause is in training, not in the mask (see below).
- The current block. Inside it, attention is still bidirectional, so a cache within the block is again approximate. Fast-dLLM v2 adds DualCache at the sub-block level for this.
Teacher forcing vs. diffusion forcing
Why must block wait for block ? Because of how BD3-LM trains. The noisy block always conditions on clean earlier blocks, the half of . This is teacher forcing: the model never sees an unfinished prefix, so at inference the prefix must be finished first.
D2F (my notes) keeps the block-causal mask and changes the training input. Every block is noisy, with mask ratios rising from left to right (). So block learns to denoise from earlier blocks that are still partly masked. This is diffusion forcing, a name borrowed from video models, where each frame gets its own noise level. At inference, D2F appends a new block once the last one is only partly decoded and decodes several blocks in one forward pass. Finished blocks are still cached exactly.
| Earlier blocks during training | At inference, block starts | |
|---|---|---|
| Block diffusion (BD3-LM, Fast-dLLM v2) | clean | when block is finished |
| D2F | noisy, less masked than block | when block is partly decoded |
| MultiTF (MBD-LMs) | clean before a group of 2–4 blocks; inside it, noisy with large gaps | when block is partly decoded |
MultiTF (my notes) sits between the two. At inference, D2F-style decoding usually has only about two active blocks, and their mask ratios are far apart: one nearly done, the next mostly masked. D2F trains on a long run of blocks whose mask ratios rise smoothly, which rarely looks like that. MultiTF post-trains a block diffusion model (LLaDA2, SDAR) on a clean prefix plus a small group of noisy blocks, with random and often large gaps between their mask ratios. With D2F’s smooth schedule instead, accuracy drops by about 5 points.
D2F gets its block-causal model by distillation from LLaDA or Dream, and not from an AR checkpoint. That is a separate choice from the forcing.
Who uses it
The larger recent dLLMs are block diffusion models converted from AR checkpoints (M07): SDAR, Fast-dLLM v2 and LLaDA 2.0. The caching side, where finished blocks form an exact cache, is P03.
References
- Arriola et al. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LM). ICLR 2025. arXiv
- Sahoo et al. Simple and Effective Masked Diffusion Language Models (MDLM). NeurIPS 2024.
- Wu et al. Fast-dLLM v2: Efficient Block-Diffusion LLM. ICLR 2026.
- Cheng et al. SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation. 2025.
- Bie et al. LLaDA2.0: Scaling Up Diffusion Language Models to 100B. 2025.
- Wang et al. Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing (D2F). 2025.
- Jin et al. Multi-Block Diffusion Language Models. 2026. arXiv
- Chen et al. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. NeurIPS 2024.