WorkJanuary 22, 2026via Vercel Blog
Testing if "bash is all you need"
Why it matters
A rigorous benchmark comparing agent design patterns (SQL vs bash vs hybrid) for structured data querying exposes a common misconception in the AI community and demonstrates how transparent evals surface tool trade-offs that matter for production deployments.
Key signals
- SQL agents achieved 100% accuracy vs bash at 52.7% on GitHub issue/PR queries
- Bash used 7x more tokens and cost 6.5x more than SQL while taking 9x longer (401s vs 45s)
- Hybrid agent (SQL + bash verification) hit 100% accuracy but doubled token cost vs pure SQL
- Performance bottlenecks: stat() calls across 68,000 files timing out at 10 seconds
- Eval corrections: 5 questions had wrong expected answers; bash agent sometimes found additional valid results
- Bash agent generated sophisticated shell chains (find, grep, jq, awk, xargs) that didn't improve task performance
- Post-mortem: filesystem agent (basic tools only) achieved 63% accuracy with better token efficiency than full bash
The hook
SQL dominates bash for agent tasks: 100% accuracy vs 53%, 6.5x cheaper, 9x faster. Here's what the eval actually revealed about agent abstraction choices.
We invited from to share how they tested the "bash is all you need" hypothesis for AI agents.Ankur GoyalBraintrust
There's a growing conviction in the AI community that filesystems and bash are the optimal abstraction for AI agents. The logic makes sense: LLMs have been extensively trained on code…