Paper notes · arXiv 2026
Multi-Block Diffusion Language Models (MBD-LMs)
A block diffusion model (M08) decodes one block at a time. The next block waits until this one is finished and its K/V are written to the cache. D2F (my notes), from the same group, lets several blocks run at once. This paper asks why that works less well than it should, and fixes two things:
- Training (§3.2, MultiTF): a short post-training run on block diffusion models such as LLaDA2-Mini and SDAR. Each training sequence is a clean prefix followed by a small group of 2–4 noisy blocks. Inside a group, each block is masked more than the one before it, by a random and often large margin.
- Inference (§3.3, Block Buffer): a fixed number of block slots per request. A new block takes an idle slot instead of making the input longer, and a finished block leaves after its K/V are cached. The input shape never changes, so the step can be replayed as a CUDA Graph.
Bar height = fraction of the block masked. Each row is one training sequence, seen from block 4.
Result on LLaDA2-Mini: 3.47 → 6.19 tokens per forward, accuracy 79.95 → 81.03, and 517 → 746 tokens/s (Tables 1 and 3).
Loading PDF… open it directly
PDF reproduced unmodified under CC BY 4.0. Highlights and notes are mine.