Into Production · P01
Caching without causality
Bidirectional attention breaks the KV cache. Mostly.
The problem
An AR model computes each position’s keys and values (K/V) once and keeps them. Causal attention means position only sees positions , and those never change, so old K/V stay exact forever. Each step computes one new position.
A dLLM unmasks a position at each step, and with bidirectional attention every position attends to it. So every K/V from the last step is stale, and an exact dLLM recomputes all of them, at every step. (Strictly, at layer 1 only the changed position’s K/V change; from layer 2 up, all of them do.) A second, quieter problem: AR knows which position comes next; a dLLM finds out only after the forward pass.
(a) AR causal attention
(b) dLLM bidirectional attention
Staleness is the main problem, and it is uneven. The prompt never changes, so its K/V barely move: cache them once and accept the small error.
The way out is empirical: stale K/V are close to fresh ones. Adjacent steps have cosine similarity near 1, decoded tokens settle, and only a few tokens change much per step. The papers differ in which stale K/V they trust, and for how long.
Reuse a stale cache (training-free)
These run on full-attention dLLMs like LLaDA and Dream as they are.
Fast-dLLM (2025, notes). Decode the answer in blocks, left to right. At the start of a block, compute K/V for the prompt and finished blocks once, then reuse them for every step inside the block (prefix cache). DualCache also caches the still-masked blocks to the right. Refresh at each block boundary. With its parallel decoding, up to 27.6× throughput on LLaDA and Dream.
dKV-Cache (NeurIPS 2025, notes). No blocks. A token’s K/V jump at the step it is decoded, then barely move. So cache a token only after it is decoded, and one step late to skip the jump; keep recomputing masked tokens; refresh everything every steps. 2–10× speedup.
dLLM-Cache (ICML 2026, notes). Caches K, V, attention and FFN outputs per layer, split by role: the prompt is refreshed rarely, the response often. Between full refreshes it recomputes only the 25% of response tokens whose value vectors changed most (V-verify). Up to 9.1× fewer FLOPs.
Elastic-Cache (ICLR 2026, notes). Makes refresh adaptive instead of periodic. When: track the decoded token that gets the most attention; if its attention pattern drifts past a threshold, refresh. Where: only from the layer that drifted upward, since deeper layers drift more; shallow layers keep their cache. Masks far to the right keep their cache too, since they get little attention. Up to 45.1× on long GSM8K generations.
Make the cache exact (change the model)
Decoding in blocks is not enough: Fast-dLLM above decodes in blocks too, and its cache is still approximate. What decides is the attention mask (M08, P03).
Block diffusion (BD3-LM, ICLR 2025, arXiv). Train with attention that is causal across blocks and bidirectional within one. A finished block never sees later blocks, so its K/V are final and the cache is exact, as in AR. It also lets the output grow block by block. Details in M08.
Fast-dLLM v2 (ICLR 2026, arXiv). Turns a pretrained AR model (Qwen2.5) into a block diffusion model with roughly a billion tokens of fine-tuning. Two cache levels: finished blocks are exact read-only context; inside the current block, DualCache handles parallel decoding. Up to 2.5× faster than the original AR model at matching quality.
Side by side
| Cached | Refreshed | Exact? | Retrain? | |
|---|---|---|---|---|
| Fast-dLLM | prompt, finished blocks (+ masked suffix) | each block | no | no |
| dKV-Cache | tokens decoded ≥ 1 step ago | every steps | no | no |
| dLLM-Cache | prompt; most of the response | prompt rarely, response often + V-verify | no | no |
| Elastic-Cache | shallow layers, far masks | when attention drifts, from that layer up | no | no |
| Block diffusion | finished blocks | never | yes | yes |
| Fast-dLLM v2 | finished blocks (+ DualCache inside) | inside the block only | across blocks | fine-tune |
The trend: training-free methods moved from fixed schedules to adaptive ones that measure drift, while new models build the block structure in so that most of the cache is exact. Reported speedups use different models, lengths and baselines, so do not compare them across rows.
References
- Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. 2025.
- Ma et al. dKV-Cache: The Cache for Diffusion Language Models. NeurIPS 2025.
- Liu et al. dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching. ICML 2026.
- Nguyen-Tri, Ranjan, Shen. Attention Is All You Need for KV Cache in Diffusion LLMs (Elastic-Cache). ICLR 2026.
- Arriola et al. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. ICLR 2025.
- Wu et al. Fast-dLLM v2: Efficient Block-Diffusion LLM. ICLR 2026.