dLLM Notes

Into Production · P01

Caching without causality

Bidirectional attention breaks the KV cache. Mostly.

4 min read

The problem

An AR model computes each position’s keys and values (K/V) once and keeps them. Causal attention means position ii only sees positions ≤i\le i, and those never change, so old K/V stay exact forever. Each step computes one new position.

A dLLM unmasks a position at each step, and with bidirectional attention every position attends to it. So every K/V from the last step is stale, and an exact dLLM recomputes all of them, at every step. (Strictly, at layer 1 only the changed position’s K/V change; from layer 2 up, all of them do.) A second, quieter problem: AR knows which position comes next; a dLLM finds out only after the forward pass.

(a) AR causal attention

KVQThecatdownsatstep tK/V computed: 1 positionThecatsat.downstep t+1K/V computed: 1 positionreuse

(b) dLLM bidirectional attention

KVQThecatsatMMMstep tK/V computed: all 5 positionsThecatsatdownMMstep t+1K/V computed: all 5 positions, againstale
Two consecutive steps, drawn after dKV-Cache's Fig. 1. AR reuses every old K/V and computes one position. In the dLLM, “sat” replaces a mask, and every position saw that mask, so last step's K/V cannot be reused exactly.

Staleness is the main problem, and it is uneven. The prompt never changes, so its K/V barely move: cache them once and accept the small error.

The way out is empirical: stale K/V are close to fresh ones. Adjacent steps have cosine similarity near 1, decoded tokens settle, and only a few tokens change much per step. The papers differ in which stale K/V they trust, and for how long.

Reuse a stale cache (training-free)

These run on full-attention dLLMs like LLaDA and Dream as they are.

Fast-dLLM (2025, notes). Decode the answer in blocks, left to right. At the start of a block, compute K/V for the prompt and finished blocks once, then reuse them for every step inside the block (prefix cache). DualCache also caches the still-masked blocks to the right. Refresh at each block boundary. With its parallel decoding, up to 27.6× throughput on LLaDA and Dream.

promptblock 0block 1block 2

12 answer tokens in 3 blocks of 4, one token per step. Switch modes and step through: the first step of each block refreshes the cache, the rest reuse it. Counts are token positions run through the model.

dKV-Cache (NeurIPS 2025, notes). No blocks. A token’s K/V jump at the step it is decoded, then barely move. So cache a token only after it is decoded, and one step late to skip the jump; keep recomputing masked tokens; refresh everything every NN steps. 2–10× speedup.

dLLM-Cache (ICML 2026, notes). Caches K, V, attention and FFN outputs per layer, split by role: the prompt is refreshed rarely, the response often. Between full refreshes it recomputes only the 25% of response tokens whose value vectors changed most (V-verify). Up to 9.1× fewer FLOPs.

Elastic-Cache (ICLR 2026, notes). Makes refresh adaptive instead of periodic. When: track the decoded token that gets the most attention; if its attention pattern drifts past a threshold, refresh. Where: only from the layer that drifted upward, since deeper layers drift more; shallow layers keep their cache. Masks far to the right keep their cache too, since they get little attention. Up to 45.1× on long GSM8K generations.

Make the cache exact (change the model)

Decoding in blocks is not enough: Fast-dLLM above decodes in blocks too, and its cache is still approximate. What decides is the attention mask (M08, P03).

Block diffusion (BD3-LM, ICLR 2025, arXiv). Train with attention that is causal across blocks and bidirectional within one. A finished block never sees later blocks, so its K/V are final and the cache is exact, as in AR. It also lets the output grow block by block. Details in M08.

Fast-dLLM v2 (ICLR 2026, arXiv). Turns a pretrained AR model (Qwen2.5) into a block diffusion model with roughly a billion tokens of fine-tuning. Two cache levels: finished blocks are exact read-only context; inside the current block, DualCache handles parallel decoding. Up to 2.5× faster than the original AR model at matching quality.

Side by side

CachedRefreshedExact?Retrain?
Fast-dLLMprompt, finished blocks (+ masked suffix)each blocknono
dKV-Cachetokens decoded ≥ 1 step agoevery NN stepsnono
dLLM-Cacheprompt; most of the responseprompt rarely, response often + V-verifynono
Elastic-Cacheshallow layers, far maskswhen attention drifts, from that layer upnono
Block diffusionfinished blocksneveryesyes
Fast-dLLM v2finished blocks (+ DualCache inside)inside the block onlyacross blocksfine-tune

The trend: training-free methods moved from fixed schedules to adaptive ones that measure drift, while new models build the block structure in so that most of the cache is exact. Reported speedups use different models, lengths and baselines, so do not compare them across rows.

References

  1. Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. 2025.
  2. Ma et al. dKV-Cache: The Cache for Diffusion Language Models. NeurIPS 2025.
  3. Liu et al. dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching. ICML 2026.
  4. Nguyen-Tri, Ranjan, Shen. Attention Is All You Need for KV Cache in Diffusion LLMs (Elastic-Cache). ICLR 2026.
  5. Arriola et al. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. ICLR 2025.
  6. Wu et al. Fast-dLLM v2: Efficient Block-Diffusion LLM. ICLR 2026.