How AI training scales
OpenAI just proved neural network training can be systematized—and it changes everything about scaling.
Why it matters
OpenAI's discovery that gradient noise scale predicts training parallelizability removes a fundamental constraint on AI scaling, suggesting batch sizes can grow far larger without diminishing returns. This reframes AI training from art to science, with direct implications for how quickly future systems can be developed.
The key facts
10 to knowGradient noise scale metric predicts parallelizability across wide range of tasks
Complex tasks have noisier gradients, enabling larger batch sizes
Finding removes one potential limit to further AI system growth
Neural network training can be rigorized and systematized rather than treated as art
Published Dec 14, 2018 (historical research from OpenAI)
Gradient noise scale predicts parallelizability of neural network training
Complex tasks have noisier gradients
Larger batch sizes becoming viable as scaling continues
Removes one potential limit to AI system growth
Published Dec 14, 2018 (6+ years old)
Go to the source
OpenAI Blogopenai.com
Publisher excerpt: We’ve discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training on a wide range of tasks. Since complex tasks tend to have noisier gradients, increasingly large batch sizes are likely to become useful in the future, removing one…