DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities
30.7%. That's the best recall rate any model achieved finding real vulnerabilities in code—and it reveals why your AI security strategy needs to mix frontier models with cheaper alternatives.
Why it matters
DeepsecBench establishes the first standardized benchmark for evaluating LLM performance on cybersecurity vulnerability detection, showing that frontier models dominate but cost-efficient alternatives (Kimi K3, Grok 4.5) are closing the gap—critical data for enterprises building AI-powered security scanning programs.
The key facts
10 to knowGPT-5.6 Sol (xhigh) ranks #1 with 35.58 score, 30.7% recall, 96.3% precision at $55.98 cost
Kimi K3 (high) ranks #8 at 17.56 score for $12.38—half the top score for ~22% of the cost
Grok 4.5 (high) ranks #10 at 15.58 score for only $5.60—best cost efficiency among top performers
Claude Opus 5 (max) ranks #2 at 32.57 score but costs $127.93—highest price without proportional performance gain
Benchmark tested 25 models across 231 human-judged vulnerabilities in 50 open-source files
Golden set kept secret (no repository, commit, or findings disclosed) to prevent training data leakage
Best model found only 71 of 231 vulnerabilities (30.7% recall)—20 of 25 runs under 20%
Production codebase scanning costs estimated at $1,200 (Kimi K3) to $5,000+ (frontier models) per full pass
Benchmark integrated with Vercel AI Gateway for multi-model routing and failover
Article published July 2026—uses future model names (GPT-5.6, Claude Opus 5) indicating speculative/forward-looking content
Go to the source
Vercel Blogvercel.com
Publisher excerpt: Last week, OpenAI evaluated two models on an exploit benchmark within an isolated sandbox. Guardrails were reduced for testing, and the models found a vulnerability in their environment, accessed the internet, and reached Hugging Face's production database. No human directed the action, but the…