dLLM Notes

Paper notes · arXiv 2026

Large Language Continuous Diffusion Models (Nemotron-Labs-Sigma)

Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Mohammad Mahdi Kamani, Yonggan Fu, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Jukić, Arash Vahdat, Morteza Mardani

Every other dLLM in these notes is discrete: noise replaces tokens with [MASK] or with random tokens (M02). Sigma is continuous: each token is mapped to a point in a 16-dimensional embedding space, Gaussian noise is added there, and DDPM or DDIM samplers denoise it, as for images. M12 covers the idea in general; this note is about the paper.

The rest is familiar: block-causal attention with blocks of 32 (M08), initialized from an AR checkpoint (M07), an exact KV cache for finished blocks (P03).

What makes the paper useful is its baseline. NLD (Nemotron-Labs-Diffusion) is a masked block diffusion model from the same group, with the same data, architecture, parameters and FLOPs. So the comparison isolates one question: continuous or discrete noise, everything else equal.

The same 13-token answer, three ways. Time is in toy units, one AR step = 1. Classic speculative decoding: a separate small drafter guesses 4 tokens one after another (0.25 each), the big model checks them in one pass. Self-speculative: the same model's diffusion head drafts a whole block of 8 from noise in one call, and its AR head checks it; each of these passes covers a whole block, so it costs more than an AR step (1.25 here; the paper measures a round at 3.0–3.5 AR steps for blocks of 32). All three output exactly the AR model's greedy text. §6, the last part of the paper.

Loading PDF… open it directly

PDF reproduced unmodified under CC BY 4.0. Highlights and notes are mine.