FrontierSeptember 11, 2026via MarkTechPost
Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize
Why it matters
HarnessDev benchmarks whether LLMs can build and evolve their own execution scaffolding (agent harnesses) from feedback. The 53% generalization failure rate on held-out tasks suggests autonomous agents hit a wall when trying to bootstrap their own infrastructure — a capability gap that matters for real-world agent reliability.
Key signals
- HarnessDev benchmark: 6 LLMs, 2,207 tasks across 5 benchmarks
- Self-built harnesses match human references on writing and ML experimentation
- Self-built harnesses trail on code and search tasks
- Only 34 of 64 evolution changes (53%) generalize to held-out tasks
- Collaborators: ByteDance Seed, SUTD, Georgia Tech, M-A-P, TokenWave.AI
- Seed score starts at 0; models construct harnesses from execution feedback
- HarnessDev benchmark: 6 LLMs tested across 5 benchmarks and 2,207 tasks
- Self-built harnesses: match human references on writing and ML experimentation; trail on code and search
- Generalization failure: only 34 of 64 (53%) evolution changes generalize to held-out tasks
- Research collaboration: ByteDance Seed, SUTD, Georgia Tech, M-A-P, TokenWave.AI
- Metric: scores runnable harness construction, not answer quality
The hook
Only 34 of 64. That's the generalization rate when LLMs try to engineer their own agent harnesses — and it reveals a hard limit on autonomous system self-improvement.
ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI introduce HarnessDev, a benchmark that scores the runnable harness a model builds rather than the answer it returns. Starting from a seed that scores 0, 6 creator LLMs construct harnesses across 5 benchmarks and 2,207 tasks, then evolve the…