Vision‑language mixture‑of‑experts still choke on high‑resolution images because the router treats every token identically, letting image patches overwhelm their expert lanes while text tokens sit idle. Geometry‑guided routing flips that script by carving out separate pathways for visual and linguistic tokens, restoring balance while maintaining comparable downstream quality.
The prevailing solution, the standard token‑level Switch auxiliary loss (Std‑Aux), only equalizes the total mixed load; as a result, large image loads can cancel against text loads, producing more than a fivefold swing in imbalance when resolution changes [1]. This hidden cancellation leaves practitioners blind to modality‑specific overloads until performance degrades.
“Across four split backbones, ReBA lowers benchmark‑input load with comparable mean task quality.” The new scheduler enforces a modality boundary and an equal‑weight per‑image instance, so each image’s patches share the same routing decision while text tokens are handled independently [1].
“ReBA lowers mean layer CV for every benchmark and backbone.” By smoothing the coefficient of variation across layers, ReBA guarantees that no single expert becomes a bottleneck, regardless of how many high‑resolution patches appear in a batch [1].
“ReBA stays below Coupled‑ImgInst at all five settings and below Std‑Aux at four settings.” Even when image resolution or tiling is aggressively varied, the geometry‑aware scheduler consistently outperforms both the coupled image‑instance baseline and the traditional auxiliary loss in raw load metrics [1].
The paper’s evaluation stops at benchmark‑level load statistics; it does not report wall‑clock speedups, memory savings, or behavior on trillion‑parameter backbones. Moreover, ReBA assumes that patches from a single image can be treated as a coherent instance—a premise that may break for heavily augmented or mixed‑image batches.
If the reported lossless balancing holds across larger deployments, practitioners should replace Std‑Aux with the ReBA scheduler in their vision‑language MoE pipelines and rerun token‑distribution profiling. The expected outcome is a uniformly utilized expert pool that scales gracefully with image resolution, freeing resources for deeper model exploration rather than firefighting overload.
