Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts
Training data pruning could cut LLM hallucinations by improving fact memorization—new research shows less data, better results.
Why it matters
Apple researchers present evidence that strategic data pruning improves how LLMs memorize factual knowledge, addressing a core limitation in knowledge-intensive applications. This suggests a paradigm shift in training efficiency: removing low-value data beats scaling up.
The key facts
10 to knowPublished at ICLR 2026 Workshop on Data Problems for Foundation Models
Formalizes fact memorization from information-theoretic perspective
Shows fact accuracy is suboptimal when training data information exceeds model capacity
Addresses hallucination and knowledge-intensive task performance
Apple Machine Learning Research authorship
ICLR 2026 Workshop on Data Problems for Foundation Models
Focus: fact memorization from information-theoretic perspective
Key finding: fact accuracy is suboptimal when training data information exceeds model capacity
Implication: strategic data pruning improves memorization and reduces hallucinations
Published by Apple Machine Learning Research
Go to the source
Apple Machine Learningmachinelearning.apple.com
Publisher excerpt: This paper was accepted at the Workshop on Navigating and Addressing Data Problems for Foundation Models at ICLR 2026. Large language models (LLMs) can struggle to memorize factual knowledge in their parameters, often leading to hallucinations and poor performance on knowledge-intensive tasks. In…