When LLM judges agree, should we believe them?
LLM judges aren't diverse thinkers—they're echoes. Amazon Science shows why consensus among AI evaluators might hide groupthink, not truth.
Why it matters
As AI labs increasingly use LLMs to judge other LLMs' outputs (evals, benchmarks, safety), correlated judge outputs—appearing to agree but reflecting training-data artifacts rather than genuine reasoning—undermine eval validity. This matters for practitioners building reliable benchmarks and for enthusiasts tracking whether capability measurements are real.
The key facts
8 to knowAmazon Science research on LLM judge correlation bias
Discounting correlated judge outputs improves eval diversity
Implication: many published benchmarks using multiple LLM judges may overstate consensus quality
Published Aug 26, 2026
Research addresses LLM judge bias and correlation in evaluation panels
Proposes discounting correlated judge outputs to preserve perspective diversity
Implications for benchmark validity and capability measurement
Methodology applicable to both frontier model evals and open-source judge-based systems
Go to the source
Amazon Scienceamazon.science
Publisher excerpt: Discounting the opinions of LLM judges with highly correlated outputs ensures that panels of judges reflect a true diversity of perspectives.