FrontierAugust 26, 2026via Amazon Science
When LLM judges agree, should we believe them?
Why it matters
As AI labs increasingly use LLMs to judge other LLMs' outputs (evals, benchmarks, safety), correlated judge outputs—appearing to agree but reflecting training-data artifacts rather than genuine reasoning—undermine eval validity. This matters for practitioners building reliable benchmarks and for enthusiasts tracking whether capability measurements are real.
Key signals
- Amazon Science research on LLM judge correlation bias
- Discounting correlated judge outputs improves eval diversity
- Implication: many published benchmarks using multiple LLM judges may overstate consensus quality
- Published Aug 26, 2026
- Research addresses LLM judge bias and correlation in evaluation panels
- Proposes discounting correlated judge outputs to preserve perspective diversity
- Implications for benchmark validity and capability measurement
- Methodology applicable to both frontier model evals and open-source judge-based systems
The hook
LLM judges aren't diverse thinkers—they're echoes. Amazon Science shows why consensus among AI evaluators might hide groupthink, not truth.
Discounting the opinions of LLM judges with highly correlated outputs ensures that panels of judges reflect a true diversity of perspectives.