Fast-dLLM, dKV-Cache and Sparse-dLLM speed up one request. dLLM-Serve asks what breaks when a GPU serves many at once, and fixes three things:
Logit Decomposition (§4.3): compute the [B, L, V] logits in chunks, so the activation peak is small and the freed memory goes to the KV pool.
Phase-Multiplexed Scheduler (§4.4): schedule per Refresh/Reuse phase, not per request.
Head-Centric Sparse KV Cache (§4.5): each head keeps its own top-k tokens, stored contiguously.
Result: 1.61–1.81× throughput on RTX 4090 and 1.60–1.74× on L40S over the best baseline, with tail latency down almost 4× under heavy load.
Logit Decomposition (§4.2–4.3), with and without. It saves memory only: the FLOPs and the outputs are the same. The memory goes to the KV pool, so more requests run at once.
One Refresh step
Withoutall logits at once
weights
With logit decompositionmax_num_logits = 2,048 (16 chunks)
weights
One worst-case Refresh step, 16 requests × 2048 tokens of LLaDA-8B on an L40S (48 GB, 16 GB of weights). The engine reserves this step's activation peak at startup; the rest after the weights is the KV pool, about 1.07 GB per 2048-token request. Rough estimates: layer activations are hidden states plus Q/K/V or the MLP's gate/up; the guard band and fragmentation are left out.
Where the spike comes from. Stack the last hidden states of all N=BL positions into H∈RN×d. The LM head multiplies them by the output embedding W∈RV×d:
Z=HW⊤∈RN×V,Mem(Z)=N⋅V⋅s=B⋅L⋅V⋅s,
with s=2 bytes in FP16. Three factors make it large:
V is much larger than every other width. Per position, the hidden state has d=4096 numbers, the MLP’s gate and up tensors have 2dff=24,576, and the logits have V=126,464. So Z is V/d≈31× the hidden states and V/2dff≈5× the largest tensor inside a layer.
A dLLM needs N=BL rows, AR only N=B. AR predicts only the next token, so Z is B×V: 16⋅126,464⋅2≈4 MB. A Refresh predicts every position, so L=2048 times more: 16⋅2048⋅126,464⋅2≈8.3 GB.
It grows with B. Memory is linear in B, so the batch that the KV pool could hold also produces the largest Z.
Why that costs requests. The profiler reserves the activation peak R for the life of the server, and the KV pool gets what is left:
R=max(Mem(Z),Alayer),#requests=⌊KV per requestG−Wweights−R⌋.
Without decomposition R≈BLVs=8.3 GB (plus the 0.27 GB hidden states), so ⌊(48−16−8.6)/1.07⌋=21. Reserving less does not help: when a full Refresh batch arrives, allocating Z fails, which is an OOM. The only safe alternative is a smaller B.
What decomposition changes. Each row of Z depends only on its own hidden state, zi=Whi, and decoding needs only two numbers per row:
Both are per row, so the rows can be computed in any grouping. Split H into chunks of C = max_num_logits rows, H=[H1;…;Hk] with k=⌈N/C⌉, and for each chunk compute Zj=HjW⊤, reduce it to (x^,c), and free it. Then:
The peak no longer depends on B or L, and the results are exactly the same, not an approximation. What remains is N×2 numbers (id and confidence), a few hundred KB. Now R=max(CVs,Alayer)≈1.9 GB, set by the MLP, so ⌊(48−16−1.9)/1.07⌋=28 requests. Chunks smaller than C≈Alayer/(Vs)≈6,000 rows save nothing more.
The two phases of §2.3, for one request (more, and how to batch them, in P08):
Q ↓ K/V →
computed this stepK/V read from cachenot computed[MASK]
Rows are queries, columns are keys and values. Refresh runs every position and rewrites the cache; Reuse runs only the active block and reads the rest from the cache.
Sparse KV (§2.4, §3.3, §4.5): which context tokens each head keeps, and what that costs in memory (the concept in P02).
Which tokens each head keeps (shade = attention weight)
What is stored per head
Toy attention weights of the current block's queries over 12 context tokens, 4 heads. Head 4 looks at the brackets, which the other heads ignore. “Kept” is the share of each head's attention that lands on tokens it still has.
The PDF's license does not allow reposting it here; page and line numbers refer to the version linked above.