Into Production · P02
Keeping fewer tokens
Most keys get almost no attention. Which ones can each head drop?
The caches in P01 decide when to recompute keys and values (K/V). This note asks which K/V to keep at all. Every cached token costs memory at every layer: for LLaDA-8B, about 0.5 MB per token, so 1 GB for a 2048-token request. If most of those tokens barely matter, keeping them wastes memory and bandwidth, and on a shared GPU it also limits how many requests fit.
Attention is sparse
Each attention head (one of the parallel attention computations in a layer, with its own , , projections; LLaDA-8B has 32 per layer) gives every query a softmax over all keys. The weights sum to 1, but most of the weight lands on a few keys and the rest get almost none. Dropping a key with weight 0.001 changes the output by about 0.1% of that key’s value.
dLLMs make this stronger. The response starts as [MASK] tokens, and every [MASK] has the same embedding, so dozens of them say almost nothing that one would not.
Picking the keys
Sparse-dLLM scores each context token by how strongly the current block’s queries match its key, per head , and max-pools the score over a small window so that a token’s neighbors survive with it:
Then it keeps the top tokens and evicts the rest, with . The retention ratio is the fraction of the context tokens kept, so the cache shrinks to times its dense size. The scores are recomputed at every Refresh (P08), when all K/V are fresh anyway.
One set, or one per head
Sparse-dLLM sums the scores over heads, , and keeps one top- set for all heads. That ranks tokens by how much heads need them on average. A head that tracks something the others ignore, such as matching brackets, loses its tokens to the vote:
Which tokens each head keeps (shade = attention weight)
What is stored per head
Dropping for real
Per-head selection keeps much more accuracy at the same (dLLM-Serve, §6.4). Still, a shared set has one advantage: every head keeps the same positions, so one gather serves all heads and the cache stays a regular tensor. Per-head sets break that. There are two ways out:
- Mask (SparseD): keep the full cache and mask out each head’s dropped tokens. Accuracy is fine, but nothing is saved: every slot is still stored and read.
- Pack (dLLM-Serve): at each Refresh, copy each head’s kept K/V into slots of a dense buffer, then forget the indices. Rows hold different tokens but have the same length.
Packing works because attention does not need the original positions. RoPE is applied to keys before they are stored, so each key already carries its position, and softmax over a set of keys does not depend on their order. Reuse steps run ordinary dense attention over each head’s slots (, and ).
| Which tokens | Stored per head | Memory | |
|---|---|---|---|
| Dense | all | ||
| Shared top- (Sparse-dLLM) | same for every head | ||
| Per-head mask (SparseD) | own set per head | ||
| Per-head packed (dLLM-Serve) | own set per head |
Since every request’s cache is a fixed times its dense size, a serving engine also knows in advance how much memory each request will take.
References
- Song et al. Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction. 2025.
- Wang et al. SparseD: Sparse Attention for Diffusion Language Models. 2025.
- Fan et al. Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving (dLLM-Serve). 2025. Notes.