dLLM Notes

Paper notes · arXiv 2025

Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving

Jiakun Fan, Yanglin Zhang, Xiangchen Li, Dimitrios S. Nikolopoulos

Fast-dLLM, dKV-Cache and Sparse-dLLM speed up one request. dLLM-Serve asks what breaks when a GPU serves many at once, and fixes three things:

Result: 1.61–1.81× throughput on RTX 4090 and 1.60–1.74× on L40S over the best baseline, with tail latency down almost 4× under heavy load.

Logit Decomposition (§4.2–4.3), with and without. It saves memory only: the FLOPs and the outputs are the same. The memory goes to the KV pool, so more requests run at once.

One Refresh step

Without all logits at once

decoder layers (6 of 32)LM head

weights

With logit decomposition max_num_logits = 2,048 (16 chunks)

decoder layers (6 of 32)LM head

weights

One worst-case Refresh step, 16 requests × 2048 tokens of LLaDA-8B on an L40S (48 GB, 16 GB of weights). The engine reserves this step's activation peak at startup; the rest after the weights is the KV pool, about 1.07 GB per 2048-token request. Rough estimates: layer activations are hidden states plus Q/K/V or the MLP's gate/up; the guard band and fragmentation are left out.

Where the spike comes from. Stack the last hidden states of all N=BLN = BL positions into H∈RN×dH \in \mathbb{R}^{N \times d}. The LM head multiplies them by the output embedding W∈RV×dW \in \mathbb{R}^{V \times d}:

Z=HW⊤∈RN×V,Mem(Z)=N⋅V⋅s=B⋅L⋅V⋅s,Z = H W^\top \in \mathbb{R}^{N \times V}, \qquad \text{Mem}(Z) = N \cdot V \cdot s = B \cdot L \cdot V \cdot s,

with s=2s = 2 bytes in FP16. Three factors make it large:

Why that costs requests. The profiler reserves the activation peak RR for the life of the server, and the KV pool gets what is left:

R=max⁡(Mem(Z), Alayer),#requests=⌊G−Wweights−RKV per request⌋.R = \max\big(\text{Mem}(Z),\ A_\text{layer}\big), \qquad \#\text{requests} = \left\lfloor \frac{G - W_\text{weights} - R}{\text{KV per request}} \right\rfloor .

Without decomposition R≈BLVs=8.3R \approx BLVs = 8.3 GB (plus the 0.27 GB hidden states), so ⌊(48−16−8.6)/1.07⌋=21\lfloor (48 - 16 - 8.6) / 1.07 \rfloor = 21. Reserving less does not help: when a full Refresh batch arrives, allocating ZZ fails, which is an OOM. The only safe alternative is a smaller BB.

What decomposition changes. Each row of ZZ depends only on its own hidden state, zi=Whiz_i = W h_i, and decoding needs only two numbers per row:

x^i=arg⁡max⁡vzi,v,ci=max⁡vsoftmax⁡(zi)v=1∑ve zi,v−max⁡uzi,u.\hat{x}_i = \arg\max_v z_{i,v}, \qquad c_i = \max_v \operatorname{softmax}(z_i)_v = \frac{1}{\sum_v e^{\,z_{i,v} - \max_u z_{i,u}}} .

Both are per row, so the rows can be computed in any grouping. Split HH into chunks of CC = max_num_logits rows, H=[H1;… ;Hk]H = [H_1; \dots; H_k] with k=⌈N/C⌉k = \lceil N / C \rceil, and for each chunk compute Zj=HjW⊤Z_j = H_j W^\top, reduce it to (x^,c)(\hat{x}, c), and free it. Then:

Mempeak=C⋅V⋅s(for C=2048: 0.52 GB),FLOPs=∑j=1k2 CdV=2NdV  (unchanged).\text{Mem}_\text{peak} = C \cdot V \cdot s \quad (\text{for } C = 2048:\ 0.52 \text{ GB}), \qquad \text{FLOPs} = \sum_{j=1}^{k} 2\,C d V = 2 N d V \ \ (\text{unchanged}).

The peak no longer depends on BB or LL, and the results are exactly the same, not an approximation. What remains is N×2N \times 2 numbers (id and confidence), a few hundred KB. Now R=max⁡(CVs, Alayer)≈1.9R = \max(CVs,\ A_\text{layer}) \approx 1.9 GB, set by the MLP, so ⌊(48−16−1.9)/1.07⌋=28\lfloor (48 - 16 - 1.9)/1.07 \rfloor = 28 requests. Chunks smaller than C≈Alayer/(Vs)≈6,000C \approx A_\text{layer} / (Vs) \approx 6{,}000 rows save nothing more.

The two phases of §2.3, for one request (more, and how to batch them, in P08):

Q ↓  K/V →

Rows are queries, columns are keys and values. Refresh runs every position and rewrites the cache; Reuse runs only the active block and reads the rest from the cache.

Sparse KV (§2.4, §3.3, §4.5): which context tokens each head keeps, and what that costs in memory (the concept in P02).

Which tokens each head keeps (shade = attention weight)

What is stored per head

Toy attention weights of the current block's queries over 12 context tokens, 4 heads. Head 4 looks at the brackets, which the other heads ignore. “Kept” is the share of each head's attention that lands on tokens it still has.

The PDF's license does not allow reposting it here; page and line numbers refer to the version linked above.