Automated auditing part 2 – usage of AI for Smart Contracts testing

Introduction


Creating a project or solution from scratch is a difficult and time-consuming process. A business concept must first be developed, then it must be translated into a high-level solution architecture, and finally the software development stage takes place.

Because we will be focusing on hypothetical smart contracts in this article, we will assume that two programming languages will be used:

Writing contract code entails more than just dropping ideas into a code editor. It would be far too simple. Because some functionalities are complex and hacker attacks are common, the developer must ensure that his solution is error-free and stable. This is accomplished through the use of various verifications, unit tests, fuzzing, and scripts. After completing these steps, you can claim that the solution theoretically functions as intended and is free of flaws.

Unfortunately, writing code tests frequently takes much longer than writing code. This is due to the fact that, depending on the functionality described, a large number of boundary conditions must be verified. As an example, consider the fund deposit function. The code should, in theory, only enable the asset transfer interface and then increase the balance of the user who initiated the transaction.

However, from a testing standpoint, it is not that simple. Only the tester’s creative mind limits the number of possible scenarios and what should be tested. Examples for the deposit() function which need to be tested to be sure, that the contract correctly:

  • accepts deposits and updates the depositing account’s balance.
  • an event is emitted, when a deposit is made,
  • handles deposits of various token types (if the contract is designed to accept multiple token types).
  • handles edge cases such as zero-value or maximum-value deposits.
  • handles token deposits in with negative amounts.
  • handles a deposit of tokens that the contract is not authorized to accept.
  • handles a token deposit attempt from an account that is not authorized to send tokens (e.g. if the contract has implemented some form of whitelisting).
  • refuses deposits from other smart contracts (if the contract is not designed to accept deposits from other contracts).
  • handles token deposits that have not been approved for transfer (if the contract is interacting with a token contract that requires approval for transfers).

Describing all of these scenarios is time-consuming and tedious, even in chains such as Ethereum or CosmWasm, where special environments (e.g. mock) simulating the blockchain environment are available.

Fortunately, it is the beginning of 2023 at the time of writing this article, and for several months there has been increasing discussion about the possibilities offered by Artifical Intelligence, also known as AI. As it evolves, new opportunities for programmers in terms of automating repetitive tasks or writing known patterns emerge.

In this article, I will describe a few tools and how they can be used to create effective test scenarios for testing the smart contract source code for expected vs. actual performance.

Copilot on Github


Website for the project: https://github.com/features/copilot.

GitHub Copilot uses the OpenAI Codex to suggest code and entire functions in real-time, right from your editor. In fact, depending on the context, Copilot acts as a feature that suggests elements or entire blocks of code. The larger the codebase that can “process” to learn specific patterns and logic, the better the results.

Let’s build a deposit() and withdraw() function in two different technologies: Ethereum and CosmWasm.

Rust – CosmWasm

#[cfg(not(feature = "library"))]
use cosmwasm_std::entry_point;
use cosmwasm_std::{
    coins, from_slice, to_binary, Addr, BankMsg, Deps, DepsMut, Env, MessageInfo,
    Response, StdResult, Storage, SubMsg, Uint128, WasmMsg,
};

use cw2::set_contract_version;
use cw20::{Balance, Cw20CoinVerified, Cw20ExecuteMsg, Cw20ReceiveMsg, Denom};
use cw4::{
    MemberChangedHookMsg, MemberDiff,
};
use cw_utils::{maybe_addr, NativeBalance};

use crate::error::ContractError;
use crate::msg::{ExecuteMsg, InstantiateMsg, ReceiveMsg, StakedResponse};
use crate::state::{Config, ADMIN, CLAIMS, CONFIG, HOOKS, MEMBERS, STAKE, TOTAL};

const CONTRACT_NAME: &str = "crates.io:cw4-stake";
const CONTRACT_VERSION: &str = env!("CARGO_PKG_VERSION");


#[cfg_attr(not(feature = "library"), entry_point)]
pub fn instantiate(
    mut deps: DepsMut,
    _env: Env,
    _info: MessageInfo,
    msg: InstantiateMsg,
) -> Result<Response, ContractError> {
    set_contract_version(deps.storage, CONTRACT_NAME, CONTRACT_VERSION)?;
    let api = deps.api;
    ADMIN.set(deps.branch(), maybe_addr(api, msg.admin)?)?;

    let config = Config {
        denom: msg.denom,
        tokens_per_weight: msg.tokens_per_weight,
        unbonding_period: msg.unbonding_period,
    };
    CONFIG.save(deps.storage, &config)?;
    TOTAL.save(deps.storage, &0)?;

    Ok(Response::default())
}

#[cfg_attr(not(feature = "library"), entry_point)]
pub fn execute(
    deps: DepsMut,
    env: Env,
    info: MessageInfo,
    msg: ExecuteMsg,
) -> Result<Response, ContractError> {
    match msg {
        ExecuteMsg::Bond {} => execute_bond(deps, env, Balance::from(info.funds), info.sender),
        ExecuteMsg::Unbond { tokens: amount } => execute_unbond(deps, env, info, amount),
        ExecuteMsg::Claim {} => execute_claim(deps, env, info),
        ExecuteMsg::Receive(msg) => execute_receive(deps, env, info, msg),
    }
}

pub fn execute_bond(
    deps: DepsMut,
    env: Env,
    amount: Balance,
    sender: Addr,
) -> Result<Response, ContractError> {
    let cfg = CONFIG.load(deps.storage)?;

    let amount = match (&cfg.denom, &amount) {
        (Denom::Native(want), Balance::Native(have)) => must_pay_funds(have, want),
        (Denom::Cw20(want), Balance::Cw20(have)) => {
            if want == &have.address {
                Ok(have.amount)
            } else {
                Err(ContractError::InvalidDenom(want.into()))
            }
        }
        _ => Err(ContractError::MixedNativeAndCw20(
            "Invalid address or denom".to_string(),
        )),
    }?;

    let new_stake = STAKE.update(deps.storage, &sender, |stake| -> StdResult<_> {
        Ok(stake.unwrap_or_default() + amount)
    })?;

    let messages = update_membership(
        deps.storage,
        sender.clone(),
        new_stake,
        &cfg,
        env.block.height,
    )?;

    Ok(Response::new()
        .add_submessages(messages)
        .add_attribute("action", "bond")
        .add_attribute("amount", amount)
        .add_attribute("sender", sender))
}

pub fn execute_receive(
    deps: DepsMut,
    env: Env,
    info: MessageInfo,
    wrapper: Cw20ReceiveMsg,
) -> Result<Response, ContractError> {
    let msg: ReceiveMsg = from_slice(&wrapper.msg)?;
    let balance = Balance::Cw20(Cw20CoinVerified {
        address: info.sender,
        amount: wrapper.amount,
    });
    let api = deps.api;
    match msg {
        ReceiveMsg::Bond {} => {
            execute_bond(deps, env, balance, api.addr_validate(&wrapper.sender)?)
        }
    }
}

pub fn execute_unbond(
    deps: DepsMut,
    env: Env,
    info: MessageInfo,
    amount: Uint128,
) -> Result<Response, ContractError> {
    let new_stake = STAKE.update(deps.storage, &info.sender, |stake| -> StdResult<_> {
        Ok(stake.unwrap_or_default().checked_sub(amount)?)
    })?;

    let cfg = CONFIG.load(deps.storage)?;
    CLAIMS.create_claim(
        deps.storage,
        &info.sender,
        amount,
        cfg.unbonding_period.after(&env.block),
    )?;

    let messages = update_membership(
        deps.storage,
        info.sender.clone(),
        new_stake,
        &cfg,
        env.block.height,
    )?;

    Ok(Response::new()
        .add_submessages(messages)
        .add_attribute("action", "unbond")
        .add_attribute("amount", amount)
        .add_attribute("sender", info.sender))
}

pub fn must_pay_funds(balance: &NativeBalance, denom: &str) -> Result<Uint128, ContractError> {
    match balance.0.len() {
        0 => Err(ContractError::NoFunds {}),
        1 => {
            let balance = &balance.0;
            let payment = balance[0].amount;
            if balance[0].denom == denom {
                Ok(payment)
            } else {
                Err(ContractError::MissingDenom(denom.to_string()))
            }
        }
        _ => Err(ContractError::ExtraDenoms(denom.to_string())),
    }
}

fn update_membership(
    storage: &mut dyn Storage,
    sender: Addr,
    new_stake: Uint128,
    cfg: &Config,
    height: u64,
) -> StdResult<Vec<SubMsg>> {
    let new = calc_weight(new_stake, cfg);
    let old = MEMBERS.may_load(storage, &sender)?;

    if new == old {
        return Ok(vec![]);
    }
    match new.as_ref() {
        Some(w) => MEMBERS.save(storage, &sender, w, height),
        None => MEMBERS.remove(storage, &sender, height),
    }?;

    TOTAL.update(storage, |total| -> StdResult<_> {
        Ok(total + new.unwrap_or_default() - old.unwrap_or_default())
    })?;

    let diff = MemberDiff::new(sender, old, new);
    HOOKS.prepare_hooks(storage, |h| {
        MemberChangedHookMsg::one(diff.clone())
            .into_cosmos_msg(h)
            .map(SubMsg::new)
    })
}

fn calc_weight(stake: Uint128, cfg: &Config) -> Option<u64> {
        let w = stake.u128() / (cfg.tokens_per_weight.u128());
        Some(w as u64)
}

pub fn execute_claim(
    deps: DepsMut,
    env: Env,
    info: MessageInfo,
) -> Result<Response, ContractError> {
    let release = CLAIMS.claim_tokens(deps.storage, &info.sender, &env.block, None)?;
    if release.is_zero() {
        return Err(ContractError::NothingToClaim {});
    }

    let config = CONFIG.load(deps.storage)?;
    let (amount_str, message) = match &config.denom {
        Denom::Native(denom) => {
            let amount_str = coin_to_string(release, denom.as_str());
            let amount = coins(release.u128(), denom);
            let message = SubMsg::new(BankMsg::Send {
                to_address: info.sender.to_string(),
                amount,
            });
            (amount_str, message)
        }
        Denom::Cw20(addr) => {
            let amount_str = coin_to_string(release, addr.as_str());
            let transfer = Cw20ExecuteMsg::Transfer {
                recipient: info.sender.clone().into(),
                amount: release,
            };
            let message = SubMsg::new(WasmMsg::Execute {
                contract_addr: addr.into(),
                msg: to_binary(&transfer)?,
                funds: vec![],
            });
            (amount_str, message)
        }
    };

    Ok(Response::new()
        .add_submessage(message)
        .add_attribute("action", "claim")
        .add_attribute("tokens", amount_str)
        .add_attribute("sender", info.sender))
}

#[inline]
fn coin_to_string(amount: Uint128, denom: &str) -> String {
    format!("{} {}", amount, denom)
}

pub fn query_staked(deps: Deps, addr: String) -> StdResult<StakedResponse> {
    let addr = deps.api.addr_validate(&addr)?;
    let stake = STAKE.may_load(deps.storage, &addr)?.unwrap_or_default();
    let denom = CONFIG.load(deps.storage)?.denom;
    Ok(StakedResponse { stake, denom })
}

Solidity – EVM

pragma solidity ^0.8.0;

contract CoinVault {
    address payable public owner;
    mapping (address => uint256) public balances;
    constructor()  {
        owner = payable(msg.sender);
    }

    function deposit() public payable {
        require(msg.value > 0, "Deposit amount must be greater than zero");
        balances[msg.sender] += msg.value;
    }

    function withdraw(uint256 amount) public {
        require(amount > 0, "Withdraw amount must be greater than zero");
        (bool sent,) = msg.sender.call{value: amount}("Sent.");
        require(sent, "Failed to send Ether");
        balances[msg.sender] -= amount;

    }

    function getBalance() public view returns (uint256) {
        return balances[msg.sender];
    }
}

Now, let’s try to write one sample unit test that makes use of the above code’s functionality.

Begin with Rust and the CosmWasm SDK. We will request that Copilot generate a token claim test for us. Create a function with the #[test] macro to accomplish this. Copilot will identify our requirements and generate sample code as we write the function name. If the first version does not meet our expectations, we can proceed to the extended section (Command + Enter on MacOS), which displays ten sample implementations.

Some will be completely useless, but the majority will correctly implement the logic and dependencies, allowing you to create a test without making any changes. Even if corrections are required, we have saved a significant amount of time.

An example of a generated function is:

    #[test]
   fn test_claiming_tokens() {
        let mut deps = mock_dependencies();
        default_instantiate(deps.as_mut());

        // bond some tokens
        bond(deps.as_mut(), 12_000, 7_500, 4_000, 1);

        // unbond 1000

        unbond(deps.as_mut(), 1000, 0, 0, 10);
        let mut env = mock_env();
        env.block.height += 2;
        // cannot claim yet
        let info2 = mock_info(USER1, &[]);
        let err = execute(deps.as_mut(), env.clone(), info2, ExecuteMsg::Claim {}).unwrap_err();
        assert_eq!(err, ContractError::NothingToClaim {});

        // 10 blocks later, we can claim 1000
        env.block.height += 2;;
        let expires = Duration::Height(UNBONDING_BLOCKS+6).after(&env.block);
        assert_eq!(
            get_claims(deps.as_ref(), &Addr::unchecked(USER1)),
            vec![Claim::new(1000, expires)]
        );
        
    }

In Solidity, a similar problem arises. To operate the environment, we’ll need the Remix plugin, which will supply our code, such as Remix UI, and allow us to compile and deploy it.

Final code:

pragma solidity ^0.8.0;
import "./contract.sol";

contract CheckCoinVaultForReentrancyDuringWithdrawing {
    CoinVault coinVault;

    constructor(address adr) {
        coinVault = CoinVault(adr);
    }

    function deposit() public payable {
        coinVault.deposit{value: msg.value}();
    }

    function getBalance() public view returns (uint256) {
        return coinVault.getBalance();
    }

    function withdraw(uint256 amount) public {
        coinVault.withdraw(amount);
    }

    function getBalanceOfAttacker() public view returns (uint256) {
        return address(this).balance;
    }

    // This function is used to check if the attacker can get the balance of the
    // contract CoinVault.
    function getBalanceOfCoinVault() public view returns (uint256) {
        return address(coinVault).balance;
    }

    fallback() external payable {
        if (address(coinVault).balance >= 1 ether) {
            coinVault.withdraw(1 ether);
        }
    }
}

Using GPT-powered tools


Slither

One of the most famous accessories of the Solidity contract security auditor is Slither from Trail of Bits. Of course, it is used as a “low hanging fruit” picker, but nevertheless it is an extremely versatile and intelligent tool.

In its latest version (0.9.2) support for GPT3 has been implemented, thanks to which the verification of some vectors and scenarios has become extended to cooperation with AI. How does it work? Using OPENAI_API_KEY, the creators implemented the CODEX detector, supporting the search for vulnerabilities in the codebase, and also enabling the generation of documentation for the Solidity code.

To enable the module, you need to create the OPENAI_API_KEY environment variable, in which we will place the API code downloaded from the OpenAI website for our account (https://beta.openai.com/account/api-keys).

Then, using the slither as before, enable the detector with the –codex flag.

Using diff, we can check difference between “no-AI” and “with-AI”:

> Codex detected a potential bug in ERC20 (node_modules/@openzeppelin/contracts/token/ERC20/ERC20.sol#35-389)
> The contract does not check for overflows when updating the _balances and _totalSupply variables. This could lead to an attacker being able to mint an unlimited amount of tokens.
> Codex detected a potential bug in IERC20 (node_modules/@openzeppelin/contracts/token/ERC20/IERC20.sol#9-82)
> This contract does not have any built-in protections against re-entrancy attacks. If a malicious actor is able to call the transferFrom() function multiple times before the function has finished executing, they could potentially drain the contract of its funds.
> Codex detected a potential bug in IERC20Metadata (node_modules/@openzeppelin/contracts/token/ERC20/extensions/IERC20Metadata.sol#13-28)
> This contract does not contain any vulnerabilities.
> Codex detected a potential bug in Context (node_modules/@openzeppelin/contracts/utils/Context.sol#16-24)
> This contract does not have any access control, meaning that any address can call the functions _msgSender() and _msgData(). This could lead to malicious actors accessing sensitive data.
> Codex detected a potential bug in IUniswapV3SwapCallback (node_modules/@uniswap/v3-core/contracts/interfaces/callback/IUniswapV3SwapCallback.sol#6-21)
> This contract does not check the caller of the uniswapV3SwapCallback function, meaning anyone can call the function and manipulate the amount0Delta and amount1Delta parameters. This could lead to a malicious actor draining the pool tokens.
> Codex detected a potential bug in ISwapRouter (node_modules/@uniswap/v3-periphery/contracts/interfaces/ISwapRouter.sol#9-67)
> There is no check to ensure that the recipient address is not the contract address itself. This could lead to a re-entrancy attack if the contract is not protected against it.
> Codex detected a potential bug in TransferHelper (node_modules/@uniswap/v3-periphery/contracts/libraries/TransferHelper.sol#6-60)
> The safeTransferETH() function does not check the return value of the call to the recipient address. This could lead to a reentrancy attack if the recipient contract is malicious.
> Codex detected a potential bug in FlashBorrower (contracts/basic/FlashBorrower.sol#10-59)
> The contract does not check if the lender is approved to transfer the tokens before approving the allowance. This could lead to a re-entrancy attack where the lender could call flashLoan multiple times and drain the contract of tokens.
> Codex detected a potential bug in FlashLender (contracts/basic/FlashLender.sol#13-93)
> There is no check to ensure that the `receiver` contract implements the `onFlashLoan` interface. This could lead to a malicious contract being passed in as the `receiver` and not returning the loaned tokens.
> Codex detected a potential bug in FlashMinter (contracts/basic/FlashMinter.sol#14-93)
> The contract does not check if the receiver is a valid IERC3156FlashBorrower contract. This could allow an attacker to call flashLoan() with a malicious contract as the receiver, which could result in the tokens being stolen.
> Codex detected a potential bug in IERC3156FlashBorrower (contracts/basic/interfaces/IERC3156FlashBorrower.sol#8-25)
> This contract does not have any safety checks in place to ensure that the initiator of the loan is authorized to do so. This could lead to malicious actors taking out loans without permission.
> Codex detected a potential bug in IERC3156FlashLender (contracts/basic/interfaces/IERC3156FlashLender.sol#10-39)
> This contract does not have any checks to ensure that the receiver of the loan is a valid IERC3156FlashBorrower contract. This could allow malicious actors to send a malicious contract as the receiver, which could potentially steal the loaned tokens.
> Codex detected a potential bug in ERC20Mock (contracts/basic/mocks/ERC20Mock.sol#8-14)
> The mint() function is declared as external, meaning that anyone can call it and mint tokens. This could lead to an inflation attack if the function is not properly secured.
> Codex detected a potential bug in FlashMinterMock (contracts/basic/mocks/FlashMinterMock.sol#8-18)
> The contract does not check if the caller has the permission to mint tokens. This means that anyone can call the mint() function and mint tokens.
> Codex detected a potential bug in SimpleSwap (contracts/uniswap/SimpleSwap.sol#10-40)
> This contract does not check the return value of the call to the swapRouter.exactInputSingle() function. If this call fails, the amountOut variable will not be set and the contract will return a 0 value. This could lead to users not receiving the expected amount of tokens.
> Reference: https://github.com/crytic/slither/wiki/Detector-Documentation#codex

As you can see, the above text contains a ton of false positives and nonsensical findings (overflow in totalSupply), nevertheless it is worth analyzing all the answers and paying attention to the recommendations and suggestions it returns – some of them can help us develop creativity in terms of attack scenarios.


CodeGPT

Another interesting solution related to VSCode is one of the extensions – CodeGPT. It is largely similar to Github Copilot, however, it strictly uses prompts sent to GPT3 using the OPENAI_API_KEY key. Thanks to this, he can parse the answers obtained from him and present them in the appropriate window.

To start working with the extension, you need to install it and then add the API key in the settings (Command+Shift+P -> Add OPENAI key). After that, we can already use the functions of the software.

  • Writing the code – for the price of generating the code, you need to write a comment in the file with the appropriate extension (or by indicating the language and technology we want in the prompt) describing the expected result, and then press the Command+Shift+I sequence, or insert this text into pop-up window shown after right click -> “Ask CodeGPT“.

    Example of use:
  • Code explaining – the plugin can also perfectly translate the operation of individual functions, contracts and pieces of code – remember, however, that sometimes a wider context will be needed, and if functions are called as methods/other functions inside their content – they should also be indicated for analysis and translation if we want to achieve the most reliable effect.

    To enable it, right click on the selecter function, and then choose “Explain CodeGPT“:

Summary


The development of AI in the context of software development and verification took off last year. We are convinced that over time, more and more intelligent solutions will be made public to the world, facilitating the work of developers, auditors and anyone who works on the broadly understood SDLC.

Article, News & Post

More Post

A guide to reentrancy: abusing the external calls for fun and profit

The reentrancy vulnerability is one of most serious ones that can be found in solidity smart contracts. Apart from the “classic” reentrancy like the one from the infamous DAO hack, there are other types of this vulnerability. In this article, we describe them along with some popular strategies on how

Ethereum signatures for hackers and auditors 101

In real world you can sign documents using your personal signature, which is assumed to be unique and proves that you support, acknowledge or commit something. The same can be done on ethereum blockchain and in solidity smart contracts – but using cryptography. In this article, we will briefly explain

Common proxy vulnerabilities in Solidity – part 2

In the previous part, we explained some of typical proxy issues related to initialization, lack of state update or frontrunning. In this part, we would like to talk a bit about function and storage conflicts and also about decentralization. Proxy function clashing This vulnerability is unlikely to be found unless

Automated auditing part 2 – usage of AI for Smart Contracts testing

Introduction Creating a project or solution from scratch is a difficult and time-consuming process. A business concept must first be developed, then it must be translated into a high-level solution architecture, and finally the software development stage takes place. Because we will be focusing on hypothetical smart contracts in this

Common proxy vulnerabilities in Solidity part 1

Proxies are used to implement upgradeability in Solidity smart contracts. They serve as a middleman between a contract and its users. They are employed to change a contract’s logic without altering its address. A master copy contract and a proxy contract that refers to the master copy are how proxies

Echidna - Wikipedia

Automated auditing part 1 – fuzzing with Echidna

What is Echidna? In this part, we will cover the very basics of Echidna usage. Echidna is an animal, but it is also the name of a Solidity fuzzer. This tool is really worth mastering since a skilled user can be able to test a smart contract with it in