Mutation Testing in Solidity With Gambit and Vertigo

Mutation testing in Solidity is, in our view, the most underrated way to test smart contracts and to find bugs in the tests themselves. Fuzzing, invariant tests, symbolic execution and static analysis get the attention.

Those are excellent tools, and you should use them alongside this one. But mutation testing gets far less interest than it deserves, and its tooling has a lot of room to grow.

This guide explains what mutation testing is, then runs it on a small contract with two tools: Gambit and Vertigo. It closes with where the tooling falls short.

What Mutation Testing Is

Mutation testing does not test your contract directly. It tests your tests, looking for vulnerabilities in the test suite.

Take a simple ticket contract with two functions, buyTicket and ownerCollect. Its test file has two tests, one per function.

contract Ticketer {
                error InvalidAmount();
                error TransferFailed();

                uint256 public constant PRICE = 1 ether;
                address public owner = msg.sender;
                mapping(address => uint256) public tickets;

                function buyTicket(uint256 amount) external payable {
                    if (amount == 0) revert InvalidAmount();
                    if (amount * PRICE != msg.value) revert InvalidAmount();
                    tickets[msg.sender] += amount;
                }

                function ownerCollect() external {
                    (bool success, ) = owner.call{value: address(this).balance}("");
                    if (!success) revert TransferFailed();
                }
            }

Running forge coverage on those two tests shows 100% line coverage. That looks well tested.

But line coverage only says which lines ran. It does not say how many bugs the suite would catch.

How a mutant works

A mutation tool finds lines in a function and changes them, one at a time. Each changed copy of the contract is a mutant.

For example, if (amount == 0) might become if (amount != 0), or simply true or false. amount * PRICE might become amount / PRICE or amount + PRICE.

Each of those changes is a bug. Your test suite is the specification of your contracts, so it should fail on every one.

If a test fails against a mutant, the mutant is caught. If every test still passes, the mutant survives, and you have found a blind spot.

Why it matters

Mutation testing measures something beyond code coverage: the bug catchability of your tests. 100% line coverage can sit right next to a critical vulnerability.

100% mutation coverage catches far more, though it still cannot catch everything. Even so, it gives you a real feedback loop on how good your tests are.

That matters because security is so hard to measure. If you cannot measure something, it is hard to improve it, and mutation testing turns part of that abstract question into a number.

Generating Mutants With Gambit

Gambit, from Certora, compiles the contract with solc and walks the parsed code. At nodes such as conditions and operators, it tries different changes.

It saves each version as an individual mutant. Every mutant carries exactly one defect.

The setup:

1. Get the binary. Download it from the Gambit releases page and put it on your path.
2. Write a JSON config. It holds a list of contracts to mutate, each with its path and any remappings needed to compile it. A config file is easier than typing the options on the command line.
3. Generate the mutants. Run gambit mutate --json with the path to your config.

On the ticket contract, Gambit produced 14 mutants in 2.67 seconds. They land in a gambit_out directory, and each one has a short comment describing its mutation.

The first mutant, for example, replaces the condition amount == 0 with true. Then every buyTicket call reverts, so the existing tests catch it easily.

The second replaces the same condition with false. That one is easier to miss.

Gambit only generates mutants. To run them, a short script loops over every mutant, copies it into the source directory, runs the test suite and records which mutants were caught.

Killing the Surviving Mutants

The first run caught 10 of the 14 mutants with just two basic tests. The two tests had Alice buy one ticket for one ether, and the owner collect that ether.

Four mutants survived. Each one pointed to a missing test:

Mutant 2 survived because no test covered a zero amount. Mutant 4 survived because no test checked that the payment had to match the price.

Mutant 12 is subtler. Every test bought exactly one ticket, so a mutant that recorded one ticket regardless of the amount looked correct.

A fuzz test closes that gap:

• Take amount as the fuzzed input.
• Assume it is not zero, which is already invalid, and keep it under 10,000 tickets for a reasonable range.
• Compute the value to send as amount times 1 ether, the ticket price.
• Deal that ether to Alice and buy as Alice.
• Assert that Alice holds amount tickets, Bob holds none, and the contract's ether balance equals the amount sent.

That test passed over 10,000 runs and killed mutant 12. One mutant still survived after unit tests, error tests and fuzzing.

Mutant 14 survived because no test made the transfer to the owner fail.

One easy way to test it: etch the ticket contract's own code, which has no receive or fallback function, at the owner's address.

With that test in place, every mutant was detected.

Line coverage had called the suite complete, yet four more test cases were needed, one of them a fuzz test, to reach full mutation coverage.

The suite now gives much stronger assurance about the code's behavior. It is also far more particular about what the code must do.

Running Vertigo

If you would rather not write a script to swap mutants in and out, use Vertigo.

The version used here is vertigo-rs, maintained by Jeffrey Scholz at RareSkills and built on the original Vertigo project.

Setup follows the README: clone the repository, run its setup.py with Python 3, then run vertigo.py run from your project.

Before running it, restore your contract to its original, unmutated version. A home-made mutation script should do this automatically when it finishes.

Never commit a mutated contract by accident. A mutant that slips into production would be a very bad day.

Vertigo detected the Foundry project on its own and logged each step. If it reports that it could not get valid results from running the tests, your test suite is failing before any mutation.

On the ticket contract, Vertigo generated six mutants. The suite caught all of them, apart from one that errored rather than running cleanly.

Gambit or Vertigo

• Vertigo is much faster to get running and has a much nicer interface. Use it to try mutation testing on a few mutants.
• Gambit with your own script produced more mutants here, 14 against 6. Use it when you want as many mutants as possible, though results depend on the contract.

Mutating Fuzz and Invariant Suites

Mutation testing is not limited to unit tests. One of the tests above was already a fuzz test, and invariant tests can join the run too.

That gives you feedback on unit, integration, fuzzing and invariant campaigns alike. It makes all of them more effective.

At Guardian, we see real promise in using it to improve the coverage of our fuzzing suites. Holding an invariant harness to high mutation coverage forces you to write genuinely smart invariants.

The result is a better covered protocol with stronger invariant testing. For proof that goes beyond testing, see formal verification for smart contracts.

Where the Tooling Falls Short

At the time of recording (2024), mutation testing was not widely used, and the tools had a lot of potential to grow. These were the main limits:

• Compile time. The ticket contract is about 15 normalized lines. With contracts of 500 or 600 lines, or many contracts, every mutant means recompiling.
• Over-compilation. Foundry tended to rebuild more than the one changed file. Its sparse configuration did not seem to help much.
• Run time. A large codebase can produce hundreds of mutants, so a full run could take hours.
• All-in-one tools. The ecosystem needed more tools like Vertigo that detect a Foundry project, generate mutants and run the suite out of the box.

Compile time is the biggest bottleneck. A simple script could also be parallelized, for example with four processes each testing its own slice of the mutants.

The payoff is a new benchmark for readiness. Before a security review or a deployment, ask where your code coverage is, and also where your mutation coverage is.

Security is fundamentally hard to measure, so any measurable indicator helps. Find the vulnerabilities in your test suite, and you will often find the ones in your contracts.

When you are getting ready for review, see how to prepare for a smart contract audit and Solidity security best practices.

How Guardian Helps

Every Vanguard audit includes an exhaustive invariant suite. Guardian writes your protocol's core safety properties as executable invariants and pressure-tests every reachable state against them.

The suite outlives the audit as a reusable regression harness for fixes and every change you ship.

Frequently Asked Questions

What is mutation testing in Solidity?

It makes many copies of a contract, each with one planted bug, such as a flipped condition or a changed operator. Your tests run against each copy, and every mutant that passes reveals a gap.

How is mutation coverage different from code coverage?

Code coverage shows which lines your tests run. Mutation coverage shows how many planted bugs your tests catch. A suite can have 100% line coverage and still miss mutants, as the ticket example shows.

Should I use Gambit or Vertigo?

Vertigo is faster to set up and has a nicer interface, so it suits a first try. Gambit, paired with your own script to run the mutants, produced more mutants in this example, so it suits a more thorough run.

Why is mutation testing slow on large codebases?

Each mutant has to be compiled and tested. Large codebases produce hundreds of mutants, and Foundry tends to recompile more than the one changed file, so a full run can take hours. Parallelizing the runs helps.

Sources

• Owen Thurm (YouTube): The #1 Most Underrated Web3 Security Tool (March 2024)
• Certora (GitHub): Gambit, mutation testing for Solidity
• RareSkills (GitHub): vertigo-rs, mutation testing for Ethereum smart contracts