dLLM Notes

Into Production · P08

Batching denoising steps

Every request is at a different noise level. Can they share a step?

4 min read

The caches in P01 split decoding into two kinds of step. For one request, that is a speedup. For many requests on one GPU, it is a scheduling problem: the two kinds cost very different amounts, and requests are never in the same one at the same time.

Two kinds of step

dLLM-Serve names them Refresh and Reuse.

  • Refresh: every position runs, and all keys and values (K/V) are recomputed and written to the cache. It fixes staleness: with bidirectional attention, every token’s K/V depend on every other token, so once a block is decoded the whole cache is out of date.
  • Reuse: only the active block runs. Its queries attend to its own fresh K/V plus the cached K/V of everything else. It fixes cost: outside the block, K/V barely change from one step to the next, so recomputing them is waste.
Q ↓  K/V →

One request, 4 prompt tokens and 3 blocks of 4, with a Refresh at the start of each block (Fast-dLLM's DualCache). Rows are queries, columns are keys and values. Click through the steps: a Refresh fills the whole grid, a Reuse only the block's rows, and most of those read cached columns.

A Reuse is an approximation: the cache misses whatever was decoded since the last Refresh. So every caching method in P01 is a choice of when to Refresh: Fast-dLLM at each block, dKV-Cache every NN steps, dLLM-Cache and Elastic-Cache when they measure enough drift.

RefreshReuse
Runsall LL positionsthe BB block positions
K/Vrecomputed, written to cacheblock fresh, rest read from cache
Whenstart of a block, or every NN stepsevery other step
Fixesstalenesscost
Bound bycomputememory bandwidth

Why they cost so differently

Every forward pass reads all the weights from GPU memory once, then does about 2 FLOPs per weight per token. So the work per byte grows with the number of tokens in the pass. Rough numbers for LLaDA-8B (16 GB of FP16 weights) on an RTX 4090 (about 165 TFLOPS, 1 TB/s):

  • Refresh, L=2048L = 2048 tokens: 2⋅8B⋅2048≈332 \cdot 8\text{B} \cdot 2048 \approx 33 TFLOP, about 0.2 s of compute. Reading the weights takes 16 ms. Compute-bound.
  • Reuse, B=32B = 32 tokens: about 3 ms of compute, but still 16 ms to read the weights, plus the cached K/V (about 1 GB per request here). Bandwidth-bound.

The crossover is around 165 tokens per pass. This is the same split as AR serving: Refresh behaves like prefill, Reuse like decode. The difference is that a dLLM request does not do one prefill and then decode; it keeps switching, once per block.

Many requests at once

A serving engine runs one packed batch per iteration, with a cap on the total number of query tokens (max_num_batched_tokens). A request in Refresh needs LL of them, a request in Reuse only BB.

Most dLLM frameworks schedule per request: a request is admitted only if its Refresh fits, and it keeps that share until it finishes. During its Reuse steps the share sits idle. dLLM-Serve schedules per phase: each iteration it charges every running request for the phase it is in right now, then admits waiting requests into whatever room is left.

budget: 4096 query tokens per iteration · Refresh 2048 · Reuse 32

Illustrative numbers: 6 requests arrive together; each is 2 blocks of one Refresh (2048 tokens) and three Reuse steps (32 tokens). Bars at the bottom are the share of the token budget used per iteration. Per request, only two fit at a time; per phase, a new request starts as soon as another drops into Reuse.

Per phase finishes the same work in about half the iterations, because the Refresh of one request now runs alongside the Reuse steps of others. AR engines do the same with chunked prefill, mixing prefill tokens into decode batches.

The logit spike

A Refresh also predicts every position, so the last layer, the LM head, turns every hidden state into a row of logits over the vocabulary: a [B,L,V][B, L, V] tensor. For 16 requests of 2048 tokens and LLaDA’s 126,464-word vocabulary, that is 16⋅2048⋅126,464⋅216 \cdot 2048 \cdot 126{,}464 \cdot 2 bytes ≈8.3\approx 8.3 GB, for one instant. An AR engine never sees this: it only needs logits for the newest token, [B,1,V][B, 1, V].

The tensor lives for one matrix multiply, but it still costs memory for good. At startup, a serving engine runs a dummy batch of the largest size it allows, measures the peak memory, and reserves that much for activations. Whatever is left after the weights and that reservation becomes the KV pool. So the 8.3 GB spike permanently takes 8.3 GB from the KV cache, and fewer requests fit.

The fix, logit decomposition, uses the fact that decoding never needs the full logits. Per token it keeps two numbers: the chosen token id and its confidence. So dLLM-Serve sets a cap, max_num_logits, and runs the LM head in chunks:

  1. take the next max_num_logits hidden states,
  2. compute their logits,
  3. argmax or sample each row, keep the ids and confidences,
  4. drop the logits and move to the next chunk.

Dropping is nothing special: the chunk’s tensor goes out of scope, and PyTorch’s allocator hands the same memory to the next chunk. The peak is one chunk, so that is all the engine reserves at startup.

max_num_logits

Logits in memory

L40S, 48 GB, reserved at startup

LLaDA-8B, 16 requests × 2048 tokens. Each row is one request's 2048 logit rows, shrunk to one strip; the hot cell is the argmax. Pick a chunk size and play: the peak follows the chunk, and the startup reservation with it. GPU split is a rough estimate that ignores other activations.

The FLOPs are the same either way; the cost is a few extra kernel launches. With 2048-token chunks, the logit reservation drops from 8.3 GB to 0.5 GB, room for about seven more 2048-token requests in the KV pool.

References

  1. Fan et al. Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving (dLLM-Serve). 2025. Notes.
  2. Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. 2025.
  3. Ma et al. dKV-Cache: The Cache for Diffusion Language Models. NeurIPS 2025.
  4. Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024.