FrontierJuly 27, 2026via Vercel Blog
DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities
Why it matters
DeepsecBench establishes the first standardized benchmark for evaluating LLM performance on cybersecurity vulnerability detection, showing that frontier models dominate but cost-efficient alternatives (Kimi K3, Grok 4.5) are closing the gap—critical data for enterprises building AI-powered security scanning programs.
Key signals
- GPT-5.6 Sol (xhigh) ranks #1 with 35.58 score, 30.7% recall, 96.3% precision at $55.98 cost
- Kimi K3 (high) ranks #8 at 17.56 score for $12.38—half the top score for ~22% of the cost
- Grok 4.5 (high) ranks #10 at 15.58 score for only $5.60—best cost efficiency among top performers
- Claude Opus 5 (max) ranks #2 at 32.57 score but costs $127.93—highest price without proportional performance gain
- Benchmark tested 25 models across 231 human-judged vulnerabilities in 50 open-source files
- Golden set kept secret (no repository, commit, or findings disclosed) to prevent training data leakage
- Best model found only 71 of 231 vulnerabilities (30.7% recall)—20 of 25 runs under 20%
- Production codebase scanning costs estimated at $1,200 (Kimi K3) to $5,000+ (frontier models) per full pass
- Benchmark integrated with Vercel AI Gateway for multi-model routing and failover
- Article published July 2026—uses future model names (GPT-5.6, Claude Opus 5) indicating speculative/forward-looking content
The hook
30.7%. That's the best recall rate any model achieved finding real vulnerabilities in code—and it reveals why your AI security strategy needs to mix frontier models with cheaper alternatives.
Last week, OpenAI evaluated two models on an exploit benchmark within an isolated sandbox. Guardrails were reduced for testing, and the models found a vulnerability in their environment, accessed the internet, and reached Hugging Face's production database.
No human directed the action, but the breach is a clear example of how much more capable malicious attackers are when equipped with powerful AI models. But defenders have the same tools, and a clear advantage: knowledge of their own codebase. Hacks are initiated from the outside, so the single best defense is finding vulnerabilities from the inside before attackers do.
Today we're releasing , a benchmark that evaluates how well different models find cybersecurity vulnerabilities in application code. For each model the report includes recall, precision, cost, and total time, and combines recall and precision into a single benchmark score. Here is a sample of model performance from the leaderboard: DeepsecBench
We built to make scanning as easy as possible. Now you can use the benchmark report to build a security scanning program that fits your budget and the complexity of your codebase, choosing the right mix of models to run and how often to run them.deepsec
DeepsecBench runs on an open-source codebase at a commit state just before a large number of vulnerabilities were fixed. We selected 50 entry-point files and built a golden set of 231 human-judged findings. Each model's score is a recall-weighted F2 (Score = 100 × 5PR/(4P+R)), weighting recall (R) twice as much as precision (P), because missed vulnerabilities will go unfixed, while false positives don't make your codebase less secure.
Findings beyond the golden set are classified by a judge model as real or false, and count for or against the model in the precision measure (recall measures only the 231 known findings).
The benchmark is run three times and the data published in the report is the median of the three runs.
The construction of the benchmark s...