CFG: amplify what the context adds. Every step makes two predictions, not one or the other:
- Conditional e^(c): the block sees the prefix c, the prompt and all finished blocks.
- Unconditional e^(∅): the same model with all of c removed, so the block sees only itself.
Then e^w=e^(∅)+w[e^(c)−e^(∅)]. The difference is what the context contributes: after “2 + 3 =”, the unconditional guess is “some number”, the conditional one leans to “5”. With w>1 that lean is amplified and the generic part subtracted, so predictions follow the context more closely. As with images, the price is diversity.
No context dropout is needed in training: with sequence packing, the first block of a training sequence often starts mid-document with no context, so the model already knows the unconditional case.
Both branches go into one forward pass with a packed mask (Fig. 5a). On H100 at batch 1 that costs 4.2% per step at blocks of 32 (App. E.9, Table 19), almost all of it in the second logit head, not attention. At batch 1 the step is memory-bound, so the extra positions are nearly free. At large batch they double the compute (P08). The paper does not measure that case.
Score temperature: sharpen, in embedding space. e^τ=zt/αt+τ1[e^−zt/αt] with τ≤1. zt/αt is the naive guess that ignores the noise. ST steps 1/τ times farther along the line from it to the model’s prediction. Since e^ is an average, inside the sphere whenever the model hesitates, this pushes it out toward a single token. The effect is that of a low sampling temperature, sampling from p1/τ, but it acts on the embedding rather than dividing the logits by T, and it only sharpens. Logit-space CFG and temperature also work, but less well (App. E.6). ST costs no extra pass.
Both knobs together beat either alone (Fig. 4): tuned alone they need extreme values (w=9 or τ=0.05); together, w=2 and τ=0.1 do better.