Improving LLM pretraining with better data organization
Amazon just found a way to cut LLM hallucinations by better organizing training data—no new hardware needed.
Why it matters
Amazon Science discovered that how you organize documents during LLM pretraining directly impacts model performance and hallucination rates. This is a foundational efficiency insight that could reshape how enterprises approach model training without infrastructure overhaul.
The key facts
11 to knowTechnique: 'Best-fit packing' adapts bin-packing algorithms to minimize document truncation
Impact: Reduces hallucination across multiple task types
Benefit: Improves LLM performance without requiring additional compute resources
Source: Amazon Science research team
Published: July 22, 2024
Best-fit packing adapts bin-packing algorithms to training data organization
Reduces unnecessary truncation of training documents during pretraining
Improves LLM performance across wide range of downstream tasks
Addresses hallucination reduction in LLM outputs
Published by Amazon Science research team
July 2024 publication
Go to the source
Amazon Scienceamazon.science
Publisher excerpt: “Best-fit packing” adapts bin-packing to avoid unnecessary truncation of training documents, improving LLM performance across a wide range of tasks and reducing hallucination.