Sparse model

Definition

A neural network where only a fraction of parameters are activated for any given input, reducing compute requirements compared to dense models of the same total size. Mixture of Experts is the most common sparse architecture.

Why it matters

Sparsity is one of the most promising paths to efficient AI. Dense models waste compute by running every parameter for every input, even when most parameters are irrelevant to the current task. Sparse models route each input to the most relevant subset of parameters, achieving large-model quality at small-model compute cost. This is significant because it partially breaks the trade-off between capability and cost. For the AI industry, sparsity suggests that the future of model scaling may not be about making every parameter bigger but about making models better at selecting which parameters to use. The architectural shift from dense to sparse could reduce inference costs by an order of magnitude.

In practice

GPT-4 and Mixtral are the most prominent sparse (MoE) models. Switch Transformer from Google demonstrated that sparse models could scale to trillions of parameters while activating only billions per token. DeepSeek's V2 and V3 models use an innovative MoE design that achieves frontier performance at a fraction of typical training cost. In benchmarks, sparse models consistently match dense models of equivalent active parameters while requiring less compute per token. The challenge is memory: even though only a subset of parameters activates per token, all parameters must be loaded into memory. This makes sparse models memory-bound rather than compute-bound, which favors hardware with high memory bandwidth.