Model WarsJuly 6, 2026via Apple Machine Learning
Path-Constrained Mixture-of-Experts
Why it matters
Apple researchers have identified a fundamental inefficiency in how Mixture-of-Experts models route tokens across layers, proposing path-constrained architectures that could significantly reduce compute waste and improve inference efficiency—a key competitive lever in the race to optimize LLM deployment.
Key signals
- N^L possible expert paths in standard MoE (N experts, L layers) remain mostly unexplored
- Tokens cluster into small fraction of paths aligned with linguistic function
- Vast majority of paths statistically unused, revealing architectural inefficiency
- Path-constrained MoE architectures proposed as optimization solution
- Published by Apple Machine Learning Research
The hook
Apple just revealed why your MoE models are wasting 99% of their paths. Here's what that means for inference cost.
Sparse Mixture-of-Experts (MoE) architectures route each token through a subset of experts at each layer independently. We propose viewing MoE computation through the lens of expert paths—the sequence of expert selections a token makes across all layers. This perspective reveals that, despite N^L possible paths for N experts across L layers, tokens in practice cluster into a small fraction of paths that align with linguistic function, yet the vast majority of paths remain unexplored, representing a statistical inefficiency. This motivates architectures that constrain the effective path space…