Image GPT
OpenAI just proved transformers work on pixels, not just text. Same model architecture. Competitive with CNNs on image classification.
Why it matters
OpenAI demonstrates that transformer architecture generalizes beyond language to vision tasks, establishing a foundation for multimodal capabilities and challenging the dominance of convolutional networks in image understanding.
The key facts
5 to knowTransformer model trained on pixel sequences generates coherent image completions and samples
Generative model features competitive with top convolutional nets in unsupervised image classification
Correlation established between sample quality and image classification accuracy
Published June 17, 2020
Historical significance: early exploration of unified architecture across modalities
Go to the source
OpenAI Blogopenai.com
Publisher excerpt: We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. By establishing a correlation between sample quality and image classification accuracy, we show that…