Testing if "bash is all you need"
SQL dominates bash for agent tasks: 100% accuracy vs 53%, 6.5x cheaper, 9x faster. Here's what the eval actually revealed about agent abstraction choices.
Why it matters
A rigorous benchmark comparing agent design patterns (SQL vs bash vs hybrid) for structured data querying exposes a common misconception in the AI community and demonstrates how transparent evals surface tool trade-offs that matter for production deployments.
The key facts
7 to knowSQL agents achieved 100% accuracy vs bash at 52.7% on GitHub issue/PR queries
Bash used 7x more tokens and cost 6.5x more than SQL while taking 9x longer (401s vs 45s)
Hybrid agent (SQL + bash verification) hit 100% accuracy but doubled token cost vs pure SQL
Performance bottlenecks: stat() calls across 68,000 files timing out at 10 seconds
Eval corrections: 5 questions had wrong expected answers; bash agent sometimes found additional valid results
Bash agent generated sophisticated shell chains (find, grep, jq, awk, xargs) that didn't improve task performance
Post-mortem: filesystem agent (basic tools only) achieved 63% accuracy with better token efficiency than full bash
Go to the source
Vercel Blogvercel.com
Publisher excerpt: We invited from to share how they tested the "bash is all you need" hypothesis for AI agents.Ankur GoyalBraintrust There's a growing conviction in the AI community that filesystems and bash are the optimal abstraction for AI agents. The logic makes sense: LLMs have been extensively trained on code,…