condition not caught by invariant/fuzzer #2851
Comments
|
I'm not sure invariant testing is the best for this kind of scenario. We have Also, does normal fuzzing even reach -- If we wanted to reach this, we could probably come up with a strategy that adds |
|
Just curious, where is this example from? I'm inclined to say we shouldn't worry too much about trying to tune the fuzzer to find this particular example—it's a bit contrived and I haven't seen much solidity code with similar logic, so you could argue it's is more of an academic issue than a practical issue. I think a more worthwhile (but also bigger) effort is ensuring foundry can match echidna's performance, which we haven't demonstrated yet. The echidna readme has a "Trophies" section showing real issues it's found, and a "Research" section reproducing examples from papers. Some of those research examples are similar to the above, but I think simpler in that there's not the All of the echidna test cases (excluding "trophies", including "research" and other tests) can be found here: https://github.com/crytic/echidna/tree/master/tests/solidity It would be great if we setup all of those test cases as part of foundry CI and make sure fuzz/invariant tests can reliably catch those |
|
@gakonst can probably provide some context and the expectations here |
|
Imo this kind of test case is what symbolic execution is more geared for. The kind of fuzzer we have is borderline a blackbox fuzzer, so it's essentially just bruteforcing its way there. We have a dictionary as well but the chance of hitting that particular number is still very low. |
|
Awesome write-up analysing this issue here https://hackmd.io/@SaferMaker/EVM-Sym-Exec |
By the way this is caught by the fuzzer now. should probably close this |
Awesome! Do you have the test handy to share, would love to have it as a regression test. Also planning to create a benchmark project (smth similar with https://github.com/grandizzy/fuzz-benchmarks/) and run daily checks. thank you! |
// SPDX-License-Identifier: MIT OR Apache-2.0
import "forge-std/Test.sol";
pragma solidity ^0.8.1;
contract Backdoor {
uint public number = 1;
function backdoor(uint256 newNumber) payable public {
uint x = newNumber - 1;
if (x == 6912213124124531) {
number = 0;
}
}
}
contract BackdoorTest is Test {
Backdoor back;
function setUp() public {
back = new Backdoor();
}
function invariantNotZero() public {
assertEq(back.number(), 1);
}
} |
|
thank you! confirm it is caught with fuzzer default settings 🎉 |
|
Closing |
Component
Forge
Have you ensured that all of these are up to date?
What version of Foundry are you on?
No response
What command(s) is the bug in?
No response
Operating System
No response
Describe the bug
The text was updated successfully, but these errors were encountered: