Sakana AI Researchers Introduce PC-ALM, a Layer-Local Alternative to Backpropagation That Trains 1000-Layer Networks
Sakana AI cracks a 40-year problem: training 1000-layer networks without backpropagation. PC-ALM matches backprop's accuracy at a fraction of the compute cost.
Why it matters
A novel training algorithm that recovers backprop-quality gradients through layer-local computation could reshape how deep networks are trained — especially relevant for edge deployment, neuromorphic hardware, and scaling laws that depend on depth.
The key facts
15 to knowPC-ALM (Predictive Coding with Augmented Lagrangian Multipliers) is a layer-local alternative to backpropagation
Recovers exact backprop gradients in linear networks; matches BP across widths 8-128 and depths up to 1000
Inference budget: T = 2L (2 iterations per layer)
Gradient cosine similarity improves from 0.604 to 0.909 on reference cell
Trains 1000-layer residual MLPs within ~2 percentage points of backprop on MNIST
MIT-licensed JAX implementation released
Authors: Jeffrey Seely and Julian Gould (Sakana AI)
Addresses biological plausibility and local-learning constraints in deep neural networks
PC-ALM (Augmented Lagrangian Predictive Coding) achieves layer-local updates without sacrificing gradient quality
Matches backprop performance across 8-128 layer depths at inference budget T=2L
Gradient cosine similarity improves from 0.604 to 0.909 in reference cell
Successfully trains 1000-layer residual MLPs within ~2 points of backprop on MNIST
MIT-licensed JAX code released publicly
Researchers: Jeffrey Seely and Julian Gould (Sakana AI)
Solves the credit-assignment problem for biologically plausible learning
Go to the source
MarkTechPostmarktechpost.com
Publisher excerpt: Sakana AI researchers Jeffrey Seely and Julian Gould introduce Augmented Lagrangian Predictive Coding (PC-ALM), a local-learning alternative to backpropagation. By attaching a Lagrange multiplier to each layer constraint, PC-ALM keeps predictive coding's layer-local updates while recovering exact…