Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize
Only 34 of 64. That's the generalization rate when LLMs try to engineer their own agent harnesses — and it reveals a hard limit on autonomous system self-improvement.
Why it matters
HarnessDev benchmarks whether LLMs can build and evolve their own execution scaffolding (agent harnesses) from feedback. The 53% generalization failure rate on held-out tasks suggests autonomous agents hit a wall when trying to bootstrap their own infrastructure — a capability gap that matters for real-world agent reliability.
The key facts
11 to knowHarnessDev benchmark: 6 LLMs, 2,207 tasks across 5 benchmarks
Self-built harnesses match human references on writing and ML experimentation
Self-built harnesses trail on code and search tasks
Only 34 of 64 evolution changes (53%) generalize to held-out tasks
Collaborators: ByteDance Seed, SUTD, Georgia Tech, M-A-P, TokenWave.AI
Seed score starts at 0; models construct harnesses from execution feedback
HarnessDev benchmark: 6 LLMs tested across 5 benchmarks and 2,207 tasks
Self-built harnesses: match human references on writing and ML experimentation; trail on code and search
Generalization failure: only 34 of 64 (53%) evolution changes generalize to held-out tasks
Research collaboration: ByteDance Seed, SUTD, Georgia Tech, M-A-P, TokenWave.AI
Metric: scores runnable harness construction, not answer quality
Go to the source
MarkTechPostmarktechpost.com
Publisher excerpt: ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI introduce HarnessDev, a benchmark that scores the runnable harness a model builds rather than the answer it returns. Starting from a seed that scores 0, 6 creator LLMs construct harnesses across 5 benchmarks and 2,207 tasks, then evolve…