Do large language models really need all those layers?
70% of attention heads are dead weight. Amazon Science just proved LLMs are massively over-engineered.
Why it matters
Amazon's research reveals that large language models contain significant redundancy—70% of attention heads and 20% of feed-forward networks can be removed without degrading performance. This challenges assumptions about LLM architecture efficiency and suggests models are undertrained, with major implications for inference cost and model compression.
The key facts
5 to know70% of attention heads can be excised with minimal performance impact
20% of feed-forward networks can be removed without effect on in-context learning
Suggests LLMs are undertrained and over-parameterized
Published by Amazon Science
Research focus: architectural redundancy in transformers
Go to the source
Amazon Scienceamazon.science
Publisher excerpt: Finding that 70% of attention heads and 20% of feed-forward networks can be excised with minimal effect on in-context learning suggests that large language models are undertrained.