How AI training scales
OpenAI's discovery that gradient noise scale predicts training parallelizability removes a fundamental constraint on AI scaling, suggesting batch sizes can grow far larger without diminishing returns. This reframes AI training from art to science, with direct implications for how quickly future systems can be developed.
December 2018
Read full story