N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?
GPT-5.4 vs Claude vs Gemini: who actually finds real security bugs in production code?
Why it matters
N-Day-Bench introduces a monthly-refreshing vulnerability discovery benchmark that tests frontier LLMs on real, uncontaminated code from GitHub security advisories—solving the critical problem of static benchmarks becoming obsolete as training data leaks render scores meaningless.
The key facts
7 to knowMonthly refresh cycle prevents training data contamination and memorization
Tests GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, GLM-5.1, Kimi K2.5
Models get 24 shell steps to explore live codebases and identify vulnerabilities
Only repos with 10k+ stars qualify; diversity filtering prevents single-repo dominance
Three-agent evaluation: Curator (builds answer key), Finder (model under test), Judge (blinded scoring)
Public traces and live leaderboard enable transparent model comparison
Addresses practical security use case (vulnerability discovery) with real-world code samples
Go to the source
Hacker Newsndaybench.winfunc.com
Publisher excerpt: N-Day-Bench tests whether frontier LLMs can find known security vulnerabilities in real repository code. Each month it pulls fresh cases from GitHub security advisories, checks out the repo at the last commit before the patch, and gives models a sandboxed bash shell to explore the codebase. Static…