Exclusive Self Attention
Apple's new attention mechanism outperforms standard Transformers on language modeling—and the gains grow with sequence length.
Why it matters
Apple Research is publishing a simple but effective architectural improvement to Transformers that could influence how foundation models handle long-context tasks. This is the kind of incremental innovation that gets adopted across the industry.
The key facts
10 to knowExclusive Self Attention (XSA) modifies standard self-attention to exclude self-position information
Tested up to 2.7B parameter models
Consistent improvements over standard attention across model sizes
Larger gains observed at longer sequence lengths
Published on Apple Machine Learning Research (apple.com/research)
Exclusive Self Attention (XSA) modification to standard self-attention
Consistent performance gains over baseline self-attention
Larger gains as sequence length increases
Published on Apple Machine Learning Research (machinelearning.apple.com)
Focuses on constraining attention to orthogonal information, excluding self-position bias
Go to the source
Apple Machine Learningmachinelearning.apple.com
Publisher excerpt: We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer’s sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token’s own value vector (thus excluding information of self…