Supabase Releases Evals: an Open Source Benchmark That Scores Claude Code, Codex and OpenCode on Real Supabase Tasks
Supabase just open-sourced a benchmark that grades Claude Code, Codex, and OpenCode on real-world database tasks — deterministic scoring, no handwaving.
Why it matters
A vendor-authored eval framework for coding agents on domain-specific tasks (database operations, Edge Functions, RLS policy fixes). Practitioners get a reproducible benchmark; enthusiasts see how capability measurement is shifting from general math/code contests to task-specific, containerized workflows.
The key facts
6 to knowOpen-source benchmark: supabase/evals (Apache 2.0)
Models evaluated: Claude Code, Codex, OpenCode
Task domains: schema building, Edge Function debugging, RLS policy fixes
Evaluation method: deterministic checks + LLM-as-judge scoring
Infrastructure: containerized stacks for reproducibility
Scope: real Supabase tasks (not synthetic)
Go to the source
MarkTechPostmarktechpost.com
Publisher excerpt: Supabase has open sourced supabase/evals, an Apache-2.0 benchmark and framework that runs coding agents including Claude Code, Codex and OpenCode against real Supabase tasks — building schemas, debugging Edge Functions, fixing RLS policies — inside containerized stacks, then scores them with…