OpenZeppelin found significant flaws in OpenAI's EVMbench, highlighting data contamination and inaccuracies in vulnerability assessment.
OpenZeppelin, a cybersecurity firm, has scrutinized OpenAI's newly launched benchmark, EVMbench, revealing significant methodological flaws and concerns over data integrity.
Introduced in mid-February in collaboration with investment firm Paradigm, EVMbench aims to evaluate how effectively AI can detect, address, and exploit vulnerabilities in smart contracts.
While OpenZeppelin praised the initiative, they subjected it to rigorous standards similar to those used for the protocols it assesses, including prominent names like Aave, Lido, and Uniswap.
Critical Findings
A key issue identified was the "contamination" of training data. EVMbench is built on a dataset of 120 vulnerabilities documented during audits from 2024 to 2025. However, the AI models tested have knowledge cutoffs as recent as August 2025, potentially recalling previously known vulnerabilities from their training data. This raises concerns about the benchmark's validity and the AI's ability to discover genuinely novel threats.
Additionally, OpenZeppelin flagged inaccuracies within the EVMbench dataset, noting that at least four vulnerabilities rated as "high risk" were actually non-functional, yet AI agents received scores for detecting these flaws incorrectly. OpenZeppelin experts stated, "This is not a subjective disagreement about severity; these are cases where the described attack simply does not work."
Experts agree that AI will be vital for enhancing blockchain security but warn that a rushed approach to implementation could jeopardize data quality and testing rigor. They emphasized the importance of ensuring that the benchmarks and data drive the tools to the same standards expected of the contracts they aim to fortify.
