Nous Research Proposes Lighthouse Attention: A Training-Only Selection-Based Hierarchical Attention That Delivers 1.4–1.7× Pretraining Speedup at Long Context
1.4–1.7× faster. Nous Research just cut long-context pretraining time in half with Lighthouse Attention—and then throws it away.
Why it matters
Nous Research demonstrates a training-efficiency breakthrough that could reshape pretraining economics for long-context models. By reducing attention complexity from O(N·S·d) to O(S²·d), Lighthouse Attention unlocks meaningful wall-clock speedups without sacrificing final model quality—a rare win in the speed-vs-performance tradeoff.
The key facts
6 to know1.40–1.69× end-to-end wall-clock speedup vs cuDNN SDPA baseline
Tested on 530M Llama-3-style model at 98K context length
Training-only mechanism; removed post-pretraining
Symmetric Q, K, V pooling across multi-resolution pyramid
Matching or lower final training loss vs baseline
Reduces attention complexity from O(N·S·d) to O(S²·d)
Go to the source
MarkTechPostmarktechpost.com
Publisher excerpt: Nous Research has published Lighthouse Attention, a selection-based hierarchical attention mechanism that wraps around standard scaled dot-product attention during pretraining and is removed afterward. Unlike prior methods such as NSA and HISA that pool only keys and values, Lighthouse pools Q, K,…