Stochastic Gradient Descent (SGD’s) Frequency Bias and How Adam Fixes It
Your LLM training pipeline has a hidden bug. SGD's frequency bias is silently degrading rare-token performance.
Why it matters
Technical deep-dive on optimization algorithm bias in LLM training. Understanding how Adam compensates for SGD's uneven gradient updates on common vs. rare tokens matters for practitioners building or fine-tuning models.
The key facts
8 to knowUneven token distribution in training data creates optimization asymmetry
Common tokens receive constant gradient updates; rare tokens receive sparse updates
Adam optimizer compensates for SGD frequency bias through adaptive learning rates
Direct relevance to model training approaches and parameter optimization strategies
SGD exhibits frequency bias: common tokens receive constant updates, rare tokens receive sparse updates
Token distribution in training data is highly skewed (small set of frequent tokens, long tail of rare tokens)
Adam optimizer addresses SGD frequency bias through adaptive learning rates
Article focuses on training methodology and optimizer selection — core to model development
Go to the source
MarkTechPostmarktechpost.com
Publisher excerpt: Modern language models are trained on data with extremely uneven token distributions. A small number of words appear in almost every sentence, while many rare but meaningful tokens occur only occasionally. This creates a hidden optimization challenge: parameters associated with common tokens…