MiniMax Sparse Attention (MSA): a Two-Branch Block-Sparse Attention Trained on a 109B-Parameter MoE With a 3T-Token Budget
28.4× reduction in attention compute. MiniMax just showed how sparse attention scales to 1M context without the capability tax.
Why it matters
MiniMax's sparse attention technique demonstrates a meaningful efficiency breakthrough in long-context inference—matching full attention quality while dramatically cutting compute at scale. This matters for builders scaling context windows on limited hardware.
The key facts
6 to knowMiniMax Sparse Attention (MSA) reduces per-token attention compute 28.4× at 1M context
Two-branch architecture: Index Branch selects Top-k blocks, Main Branch attends only to selected blocks
Built on Grouped Query Attention (GQA)
Trained on 109B-parameter MoE model with 3T-token budget
Matches GQA downstream benchmark performance
Published June 17, 2026
Go to the source
MarkTechPostmarktechpost.com
Publisher excerpt: MiniMax released MSA, a sparse attention built on Grouped Query Attention. A lightweight Index Branch selects Top-k key-value blocks per query and GQA group; the Main Branch attends only to those blocks. It matches GQA on downstream benchmarks while reducing per-token attention compute 28.4× at 1M…