OpenAI and Paradigm have launched EVMbench, a benchmark to evaluate AI agents in detecting and exploiting smart contract vulnerabilities.
OpenAI, in partnership with Paradigm, has unveiled EVMbench, a new benchmark aimed at assessing AI agents' proficiency in detecting, fixing, and exploiting vulnerabilities within smart contracts.
This tool comprises 120 selected vulnerabilities from 40 different audits, primarily sourced from open code analysis platforms. Additionally, it features attack scenarios derived from the security assessment of Tempo, a Layer 1 blockchain network developed by Stripe and Paradigm, catering to efficient transactions in stablecoins.
The inclusion of payment smart contracts through the Tempo integration highlights an area poised for growth in the use of stablecoins and AI technology.
“Smart contracts safeguard crypto assets valued at over $100 billion. As AI capabilities in coding evolve, it is crucial to evaluate their performance in real-world applications and to promote AI's role in enhancing security through auditing existing protocols,” the announcement emphasized.
For the benchmark’s environment, OpenAI has modified existing exploits and scripts to ensure their real-world application.
EVMbench assesses three main capabilities:
- Detection of vulnerabilities;
- Fixing of identified issues;
- Exploitation of weaknesses to potentially siphon funds.
AI Model Evaluation Results
OpenAI has evaluated its advanced AI models across these capabilities. In the exploitation category, the GPT-5.3-Codex model achieved a notable success rate of 72.2%, compared to GPT-5's 31.9%. However, the models showed less effectiveness in detecting and patching vulnerabilities, indicating persistent challenges in identifying and resolving many issues.
Notably, AI agents often halt after detecting a single vulnerability instead of performing comprehensive audits. Moreover, in Patch mode, these agents face difficulties in addressing subtle issues without affecting the smart contract's overall functionality.
“EVMbench does not encompass the entire complexity of smart contract security. While the benchmark scenarios are significant, many protocols undergo more stringent audits, which may present greater challenges for exploitation,” OpenAI noted.
In a related development, Microsoft launched a testing environment for AI agents in November 2025, which highlighted vulnerabilities in contemporary digital assistants.
