Nous Research Releases Token Superposition Training to Speed Up LLM Pre-Training by Up to 2.5x Across 270M to 10B Parameter Models
2.5x faster. Nous Research just cut LLM pre-training time in half with Token Superposition Training—no architecture changes needed.
Why it matters
Training efficiency breakthroughs directly impact model development velocity and compute economics. A 2.5x speedup at matched FLOPs across multiple scales (270M-10B) could reshape pre-training ROI for labs and enterprises building proprietary models.
The key facts
5 to knowToken Superposition Training (TST) reduces wall-clock training time by up to 2.5x at matched FLOPs
Two-phase approach: Phase 1 averages contiguous token embeddings into bags; Phase 2 reverts to standard next-token prediction
No changes required to model architecture, tokenizer, optimizer, or inference-time behavior
Validated at 270M, 600M, 3B dense, and 10B-A1B MoE parameter scales
Source: Nous Research via MarkTechPost
Go to the source
MarkTechPostmarktechpost.com
Publisher excerpt: Nous Research releases Token Superposition Training (TST), a two-phase pre-training method that cuts wall-clock training time by up to 2.5x at matched FLOPs by averaging contiguous token embeddings into bags during Phase 1 and reverting to standard next-token prediction in Phase 2 — without…