Learning to play Minecraft with Video PreTraining
OpenAI trained a neural network to play Minecraft using only video—no game API access. It learned to craft diamond tools in 24,000 actions. Here's why that matters for agents.
Why it matters
Video PreTraining (VPT) demonstrates a novel approach to training generalist agents on unlabeled human behavior data, moving toward AI systems that can operate across multiple domains using native human interfaces rather than structured APIs. This represents progress on the foundational capability gap: teaching models to learn from observation and transfer that learning to novel tasks.
The key facts
6 to knowTrained on massive unlabeled video dataset of human Minecraft gameplay
Fine-tuned with small amount of labeled contractor data
Model learns to craft diamond tools—task requiring 20+ minutes for proficient humans (24,000 actions)
Uses native human interface (keypresses and mouse movements)
Positioned as step toward general computer-using agents
Published June 23, 2022 by OpenAI
Go to the source
OpenAI Blogopenai.com
Publisher excerpt: We trained a neural network to play Minecraft by Video PreTraining (VPT) on a massive unlabeled video dataset of human Minecraft play, while using only a small amount of labeled contractor data. With fine-tuning, our model can learn to craft diamond tools, a task that usually takes proficient…