Learning from human preferences
OpenAI and DeepMind just solved a core AI safety problem: teaching systems what humans actually want, not what we think we want.
Why it matters
A foundational safety approach—learning human preferences instead of hand-coded reward functions—addresses a critical gap in AI alignment that remains central to safety debates today. This represents early-stage academic/safety governance work with long-term implications for how AI systems are trained.
The key facts
9 to knowCollaboration between OpenAI and DeepMind safety teams
Algorithm learns human preferences from binary comparisons (A vs B)
Addresses goal specification problem: proxies for complex goals can cause dangerous behavior
Published June 2017 (foundational RLHF-adjacent research)
Focus on safety governance and alignment methodology
Algorithm learns human preferences from pairwise behavior comparisons
Removes need for manually-written goal functions
Published June 2017 — foundational work in RLHF/preference learning
Directly addresses specification gaming and reward misalignment risks
Go to the source
OpenAI Blogopenai.com
Publisher excerpt: One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind’s safety team, we’ve…