ChatGPT can’t beat human smart contract auditors yet: OpenZeppelin’s Ethernaut challenges

cointelegraph.com:

While generative artificial intelligence (AI) is capable of doing a vast variety of tasks, OpenAI’s ChatGPT-4 is currently unable to audit smart contracts as effectively as human auditors, according to recent testing.

In an effort to determine whether AI tools could replace human auditors, blockchain security firm OpenZeppelin’s Mariko Wakabayashi and Felix Wegener pitted ChatGPT-4 against the firm’s Ethernaut security challenge. 

Although the AI model passed a majority of the levels, it struggled with newer ones introduced after its September 2021 training data cutoff date, as the plugin enabling web connectivity was not included in the test.

Ethernaut is a wargame played within the Ethereum Virtual Machine consisting of 28 smart contracts — or levels — to be hacked. In other words, levels are completed once the correct exploit is found.

According to testing from OpenZeppelin’s AI team, ChatGPT-4 was able to find the exploit and pass 20 of the 28 levels, but did need some additional prompting to help it solve some levels after the initial prompt: “Does the following smart contract contain a vulnerability?”

In response to questions from Cointelegraph, Wegener noted that OpenZeppelin expects its auditors to be able to complete all Ethernaut levels, as all capable authors should be able to.

While Wakabayashi and Wegener concluded that ChatGPT-4 is currently unable to replace human auditors, they highlighted that it can still be used as a tool to boost the efficiency of smart contract auditors and detect security vulnerabilities, noting:

When asked whether a tool that increases the efficiency of human auditors would mean firms like OpenZeppelin would not need as many, Wegener told Cointelegraph that the total demand for audits