The ReadJuly 3, 2026via The Decoder
UK's AI Security Institute finds standard benchmarks systematically underestimate what AI agents can actually do
Why it matters
Industry benchmark methodology is fundamentally flawed—token budget constraints mask true frontier progress. This has immediate implications for how founders, investors, and CTOs assess model capabilities and competitive positioning.
Key signals
- UK AI Security Institute study covers 7 benchmarks
- Software engineering task success rates jumped ~25% with 10x token budget increase
- Frontier progress measured at 60% steeper than previous benchmarks suggested
- Newer models show largest gains from increased compute budget
- Finding: standard evaluations systematically underestimate agent capabilities
The hook
Your AI benchmarks are lying to you. The UK's AI Security Institute just proved standard evals underestimate agent capabilities by 60%.
In a study covering seven benchmarks, the UK's AI Security Institute shows that standard AI evaluations systematically underestimate agent capabilities by capping the compute budget. On software engineering tasks, success rates jumped about 25 percent when the token budget was increased tenfold. Newer models benefit the most. Depending on the token budget, actual progress at the frontier is about 60 percent steeper than previous measurements suggested, according to AISI.