EVMbench Explained: What It Says About AI Auditors
Quick answer. EVMbench is OpenAI and Paradigm's open benchmark testing AI agents on 120 high-severity bugs from 40 audits. The paper's best scores were 45.6% on Detect (Claude Opus 4.6), and 41.5% on Patch and 72.2% on Exploit (GPT-5.3-Codex).
Video transcript
The top agent on EVMbench exploited 72.2 percent of bugs but patched only 41.5 percent. That gap is easy to misread.
OpenAI and Paradigm's EVMbench tests AI agents on 120 high-severity bugs from 40 audits, mostly Code4rena contests.
Detect means writing an audit report. Patch means fixing the code. Exploit means draining funds on a local Anvil chain.
Claude Opus 4.6 led Detect at 45.6 percent. GPT 5.3 Codex led both Patch and Exploit. Each agent ran three times.
But Exploit uses only 24 of Patch's 45 bugs, and intervals are wide. With a mechanism hint, GPT 5.2's Patch score more than doubled.
The paper is candid about its limits. Detect only credits known bugs, false positives cost nothing, and Exploit runs on a blank local chain.
A March 2026 re-evaluation on 22 newer incidents found zero profitable end-to-end exploits. OpenZeppelin said top agents had likely seen the bug reports during pretraining.
Use EVMbench to calibrate claims, not to pick a vendor. Ask whether the data was public before training, how many findings were false, and who verifies them.
Guardian's Vanguard audit pairs expert manual review with an independent team driven by Helix, our AI auditor. Only human-verified findings reach the report. Guardian has not published a Helix benchmark.
Read the full article for every score and its limits, or get a Vanguard quote at guardian.security.
EVMbench is a public test of whether AI agents can audit smart contracts. OpenAI and Paradigm released it on February 18, 2026, with a paper co-written by OtterSec.
Its headline numbers travel fast: 72.2% of bugs exploited, 41.5% patched. Read alone, they suggest AI attacks better than it defends.
This guide covers what the benchmark measures, how to read the scores, and what they do not tell a team choosing an audit.
What EVMbench Measures
EVMbench asks one question: can an agent find, fix and exploit fund-draining bugs in real codebases?
The dataset is 120 high-severity vulnerabilities from 40 audits:
- Source. Mostly Code4rena audit contests, plus several scenarios from the security audit of Tempo, a payments blockchain.
- Filter. Solidity bugs that can directly lose user or platform funds, found from July 2023 on.
- Size. Scopes run from 106 to 10,108 lines of code, with a median of 1,296.
- Review. A final quality pass was done with OtterSec, a smart contract security firm.
The agent gets what a human auditor got: the code, the scope, known issues and the sponsor's hints on where to look.
It works in an isolated container with internet access and web tools disabled. A separate grader container, which the agent cannot reach, scores the work.
Detect, Patch and Exploit Modes
Each mode tests a different skill, and each is graded differently.
| Mode | The agent must | How it is graded | Bugs |
|---|---|---|---|
| Detect | Write an audit report | A GPT-5 judge checks recall of the known bugs | 120 |
| Patch | Fix the code | Original tests pass; hidden exploit tests fail | 45 |
| Exploit | Drain funds on a local chain | Transactions replayed; balances checked by script | 24 |
Detect also gives a dollar score: the median Code4rena payout for each bug found. The maximum is $218,434.
Exploit is the most realistic mode. The agent gets an RPC endpoint, a funded key and contract addresses, then builds and sends the attack itself.
The chain is a fresh local Anvil instance, behind an RPC gatekeeper that blocks cheat methods. A Rust tool then replays the agent's transactions in the grader.
Headline Scores
Each agent ran three times at its highest reasoning setting, in its developer's own tool: Codex CLI, Claude Code or Gemini CLI.
| Agent | Detect | Patch | Exploit |
|---|---|---|---|
| GPT-5.3-Codex (Codex CLI) | 39.2% | 41.5% | 72.2% |
| Claude Opus 4.6 (Claude Code) | 45.6% | 25.9% | 61.1% |
| GPT-5.2 (Codex CLI) | 39.2% | 39.3% | 62.5% |
| Claude Opus 4.5 (Claude Code) | 36.1% | 21.5% | 50.9% |
| Gemini 3 Pro (Gemini CLI) | 20.8% | 10.4% | 36.1% |
| GPT-5 | 23.3% | 20.0% | 31.9% |
| OpenAI o3 | 10.6% | 14.8% | 18.1% |
In dollars, Claude Opus 4.6 earned the most in Detect: an average of $37,825 of the $218,434 available.
Paradigm frames the trend. When the project began, top models could exploit "less than 20%" of these bugs.
Reading the exploit vs patch gap
The 72.2% vs 41.5% gap is easy to misread.
- Different sets. Exploit uses 24 bugs chosen from Patch's 45.
- Wide intervals. GPT-5.3-Codex's 72.2% has a 95% interval of 56.9% to 86.1%.
- Discovery first. Given a hint naming the broken mechanism, GPT-5.2's Patch score more than doubled.
The authors suggest discovery is "the primary bottleneck in many repos." Once agents know what is broken, they can usually fix it.
The tool around the model matters too. GPT-5.2 scored higher in all three modes in Codex CLI than in the basic OpenCode loop.
Limits of the Benchmark
The paper is direct about what it cannot show:
- Not a live-bug rate. Scores do not mean that share of live blockchain bugs is exploitable.
- Known bugs only. Detect cannot credit valid bugs missing from the original report.
- No noise penalty. False positives cost the agent nothing.
- Small sets. Only 45 Patch and 24 Exploit tasks.
- Blank chain. Exploit tasks have no forking, chain history or cross-chain calls, and no complex timing.
Its own summary: EVMbench is "best read as measuring depth of review under constraints, not just the ability to find a bug."
Outside reviews
OpenZeppelin said top agents had likely seen the benchmark's vulnerability reports during pretraining, Cointelegraph reported on March 3, 2026.
It also found at least four bugs labeled high severity "where the described exploit doesn't work."
A March 11, 2026 re-evaluation by researchers from Zhejiang University and BlockSec tested agents on 22 real incidents from after mid-February 2026.
- The best agent, Claude Opus 4.6, detected 65% of graded incidents.
- No agent produced a profitable end-to-end exploit on any of the 22.
- On EVMbench's own tasks, the exploit leader changed to Claude Sonnet 4.6.
- The open-source OpenCode scaffold beat the model makers' own tools in five of six comparisons.
What It Means for Audit Buyers
EVMbench measures agents, not audit firms. Use it to calibrate claims, not to pick a vendor.
What it tells you
- The best single agent found under half of the known high-severity bugs.
- Agents can score well on an audit "while still missing critical vulnerabilities in the same codebase."
- Exploit skill is real, and the authors warn that better agents raise the attack surface.
What it does not tell you
- How a tool does on new code it has never seen.
- How much noise a tool produces.
- How any vendor's product performs in your codebase.
Questions to ask any vendor
- Which benchmark, and was its data public before the model's training cutoff?
- Who judged a match, and how many reported findings were false?
- Which model and tool ran, and how many times?
- Was it tested on code written after its cutoff?
- Does a named human verify every finding before the report?
For each tool's published evidence, see the best AI tools for smart contract auditing. If your team writes code with AI, read whether AI-generated smart contracts are safe.
How Guardian Uses AI
Vanguard, Guardian's flagship audit, puts two independent teams on one scope. Team A uses expert manual review and an invariant fuzzing suite.
Team B is driven by Helix, Guardian's AI auditor. The Vanguard page says it searches the entire codebase, "spending billions of tokens and several days across hundreds of parallel agents."
Only human-verified findings reach the report. Guardian has not published a Helix benchmark.
Our explainer on the hybrid AI and human audit model covers why a second independent team is another chance at what the first one missed.
Frequently Asked Questions
What is EVMbench?
An open benchmark from OpenAI and Paradigm, released February 18, 2026. It tests AI agents on 120 high-severity bugs from 40 audits, mostly Code4rena contests, in three modes: detect, patch and exploit.
Which model scored highest on EVMbench?
In the paper, Claude Opus 4.6 in Claude Code led Detect at 45.6%. GPT-5.3-Codex in Codex CLI led Patch at 41.5% and Exploit at 72.2%. A March 2026 re-evaluation found that rankings shift across setups and datasets.
Why do AI agents score higher on exploit than patch?
The paper does not explain the gap directly. Exploit uses 24 bugs chosen from Patch's 45, and the intervals are wide. The authors suggest discovery is the main bottleneck: given a mechanism hint, GPT-5.2's Patch score more than doubled.
Does EVMbench show AI can replace a smart contract audit?
No. Its authors say the scores do not show what share of live bugs is exploitable, and Detect does not penalize false positives. A re-evaluation on 22 newer incidents found no profitable end-to-end exploit.
Sources
- OpenAI, Paradigm and OtterSec: EVMbench, Evaluating AI Agents on Smart Contract Security (February 18, 2026)
- Paradigm: Introducing EVMbench (February 18, 2026)
- GitHub: paradigmxyz/evmbench, companion interface to the Detect evaluation
- Cointelegraph: OpenZeppelin audits OpenAI's EVMbench, finds data contamination (March 3, 2026)
- arXiv: Peng, Wu and Zhou, Re-Evaluating EVMBench, Are AI Agents Ready for Smart Contract Security? (March 11, 2026)
- Guardian: Vanguard
Put AI and human auditors on your code
Vanguard pairs expert manual review and invariant fuzzing with Helix, and only human-verified findings reach the report.