Parallax: A Parameterized Local Linear Attention That Keeps Softmax and Adds a Learned Covariance Correction Branch
Parallax doubles arithmetic intensity while keeping softmax. Here's why that matters for your model's inference cost.
Why it matters
A new attention mechanism improves computational efficiency and perplexity across model scales (0.6B–1.7B), addressing a core constraint in LLM deployment: the speed-quality tradeoff.
The key facts
10 to knowParallax replaces per-query solver with learned projector
2x arithmetic intensity improvement over LLA
Perplexity gains at 0.6B and 1.7B parameter scales
Preserves softmax (backward compatible)
Adds learned covariance correction branch
Parallax replaces LLA per-query solver with learned projector
2x arithmetic intensity improvement claimed
Perplexity improvements at 0.6B and 1.7B parameter scales
Maintains softmax while adding learned covariance correction
Published on MarkTechPost (academic/technical venue)
Go to the source
MarkTechPostmarktechpost.com
Publisher excerpt: Parallax replaces LLA's per-query solver with a learned projector, doubling arithmetic intensity and improving perplexity at 0.6B and 1.7B.