Implicit semantic registers let you drop attention heads safely; pruning heads that attend most strongly to prompt tokens reduces attention FLOPs by about 20% and incurs only a modest ~1.4‑point drop in GenEval scores. The paper reports that pruning heads attending most strongly to prompt tokens removes roughly 20% of attention FLOPs and leads to only a small ~1.4‑point decrease in GenEval, indicating that a substantial fraction (around 20–30%) of heads can be removed with limited impact., turning head‑pruning into a cheap inference hack.
Before this work, pruning strategies for diffusion transformers focused on weight magnitude, gradient saliency, or costly fine‑tuning loops, and they treated all heads as equally essential. No prior analysis had identified a subset of tokens that could serve as internal “registers” to absorb semantics while freeing the rest of the network.
Structural template tokens act as dominant image‑to‑text attention sinks and causally maintain object identity inside the DiT, acting as implicit semantic registers [1]. This behavior persists regardless of prompt complexity or language, meaning the register function is robust across varied inputs.
A training‑free rule that prunes the heads attending most strongly to prompt tokens cuts roughly 20 % of joint‑attention FLOPs while GenEval accuracy drops by only 1.4 points [1]. The pruning operation requires no additional data or retraining; it simply ranks heads by their prompt‑token attention strength and prunes the subset that attend most strongly to those tokens (approximately a quarter of heads in the authors' experiments).
The result is limited to text‑to‑image diffusion transformers evaluated on the GenEval benchmark, so its applicability to other modalities or larger language models remains unproven. Moreover, the study does not explore how pruning interacts with downstream fine‑tuning or with alternative quality metrics beyond GenEval, leaving open whether the same head budget holds under different deployment constraints.
Deployments of large diffusion transformers may consider applying the training‑free pruning rule, which can cut about 20% of attention FLOPs with minimal loss in visual fidelity, rather than adopting a blanket 25% head‑pruning default.
