Fine-tuning GPT-2 from human preferences
OpenAI fine-tuned GPT-2 on 60k human labels. Here's what the model learned that surprised them.
Why it matters
OpenAI demonstrates early RLHF methodology at scale, revealing how human feedback shapes model behavior—and the gap between stated preferences and actual labeler behavior. A foundational technique that would become critical to modern LLM alignment.
The key facts
11 to know774M parameter GPT-2 fine-tuned using human feedback
Summarization task required 60k human labels
Simpler style tasks required only 5k labels
Models learned to copy sentences wholesale from input—labeler preference diverged from researcher intent
Motivation: advancing safety techniques via RLHF for 'machines talking to humans'
Published September 2019—early RLHF exploration pre-ChatGPT era
774M parameter GPT-2 fine-tuned on human feedback
Simpler text-completion tasks required only 5k labels
Models matched labeler preferences but not always accuracy (e.g., copying wholesale from input)
Research framed as pathway to extracting human values at scale
Published September 2019 — early exploration of preference-based training
Go to the source
OpenAI Blogopenai.com
Publisher excerpt: We’ve fine-tuned the 774M parameter GPT-2 language model using human feedback for various tasks, successfully matching the preferences of the external human labelers, though those preferences did not always match our own. Specifically, for summarization tasks the labelers preferred sentences copied…