Exphormer: Scaling transformers for graph-structured data
Google just solved the graph transformer bottleneck. Exphormer scales to 170K-node datasets—10x larger than before.
Why it matters
Google Research has released Exphormer, a sparse attention framework that eliminates the quadratic computational bottleneck limiting graph transformers to small datasets. This breakthrough enables transformers to scale to graphs with 170K+ nodes, opening new applications in molecular modeling, social networks, and citation analysis.
The key facts
9 to knowExphormer scales to 170K-node graphs (ogbn-arxiv dataset with 1.1M edges)
Previous graph transformer limit: few thousand nodes
State-of-the-art on 4 of 5 Long Range Graph Benchmark datasets at publication
Uses d-regular expander graphs to reduce interaction graph edges from O(n²) to O(n)
Presented at ICML 2023
Implementation available on GitHub
PascalVOC-SP F1 score: 0.398 vs 0.375 (standard GraphGPS)
COCO-SP F1 score: 0.346 vs 0.341
PCQM-Contact MRR: 0.364 vs 0.334
Go to the source
Google Research Blogblog.research.google
Publisher excerpt: Posted by Ameya Velingker, Research Scientist, Google Research, and Balaji Venkatachalam, Software Engineer, Google Graphs, in which objects and their relations are represented as nodes (or vertices) and edges (or links) between pairs of nodes, are ubiquitous in computing and machine learning (ML).…