Fuzzing MLIR compilers with Custom Mutation Synthesis, Ben Limpanukorn, Jiyuan Wang, Hong Jin Kang, Eric Zitong Zhou, Miryung Kim, 47th International Conference on Software Engineering (ICSE '25) 12 pages https://arxiv.org/abs/2404.16947

This artifact contains the implementation of SynthFuzz and the scripts required to reproduce the results presented in the paper.

We would like to apply for the following badges:

• Available: This artifact has been made permanently available for retrieval at the following link: https://figshare.com/s/b96ea4a64f6c6a0ece12
• Functional: We have included and documented all components required to exercise the package and reproduce the results in our paper. Please see the following Usage section in the README for more information.
• Reusable: The SynthFuzz algorithm is implemented as an extension of Grammarinator and can be re-used and re-purposed for fuzzing other domains as discussed in the Generalizability section of the paper. A usage example is provided under the example directory and described below in the Usage: Example section.

This artifact can be obtained on FigShare: https://figshare.com/s/b96ea4a64f6c6a0ece12 or on GitHub at: https://github.com/UCLA-SEAL/SynthFuzz

A preprint has been made available at: https://arxiv.org/abs/2404.16947

This artifact was tested on a machine with an AMD Ryzen 2950X CPU with 32 GB of RAM.

Before running this artifact, please install Docker (Installation Instructions). Please also ensure that this artifact is extracted to a directory whose absolute path does not contain spaces.

1. Build the docker image by running ./docker/build.sh
2. Start the container by running ./docker/run_default.sh
3. Enter the container by running ./docker/attach.sh. All commands after this point should be run inside the container.

An example script and data has been provided under the example directory to demonstrate how to use SynthFuzz.

SynthFuzz is implemented as an extension to Grammarinator and can be used in the same fashion.

To run this example, simply run the example script:

cd example
./example.sh

This script will execute the follwoing commands:

Firstly, the grammar is processed to produce a test generator. For SynthFuzz, this step also generates a insert_patterns.pkl file which is used by the fuzzer to determine which production rules contain quantifiers.

python -m mlirmut.synthfuzz.process mlir_2023.g4 --rule start_rule -o mlirgen

Then, using Grammarinator, the inputs are parsed using the grammar to generate parse-trees which are used as seed inputs for the fuzzer:

grammarinator-parse \
    -r start_rule \
    -i inputs/*.mlir \
    -o trees \
    mlir_2023.g4

Finally, the fuzzer can be used to generate inputs:

python -m mlirmut.synthfuzz.generate \
    mlir_2023Generator.mlir_2023Generator \
    -r start_rule \
    -d 100 \
    -o outputs/%d.mlir \
    -n 10 \
    --sys-path mlirgen \
    --population trees \
    --insert-patterns mlirgen/insert_patterns.pkl \
    --mutation-config mutation_config.toml \
    --keep-trees \
    --no-generate --no-recombine --no-mutate \
    --k-ancestors=4 --l-siblings=4 --r-siblings=4

For more information regarding SynthFuzz-specific command line options, run: python -m mlirmut.synthfuzz.generate --help

The post-processed branch and dialect pair coverage has been included with this artifact for convenience. If you would like to reproduce the results from scratch delete the data directory and follow the directions in the Running Experiments and Collecting Coverage From Scratch section before continuing with this section. Note that running the experiments from scratch may take several days depending on your machine.

All commands should be run inside the Docker container.

cd /synthfuzz
python figures-tables/coverage.py

All commands should be run inside the Docker container.

cd /synthfuzz
python figures-tables/diversity.py

All commands should be run inside the Docker container.

cd /synthfuzz
python figures-tables/ablation-context.py

All commands should be run inside the Docker container.

cd /synthfuzz
python figures-tables/ablation-params.py

All commands should be run inside the Docker container. This section is only required if you would like to re-generate the data directory from scratch.

1. Compile each subject program:

cd /synthfuzz/eval
# build mlir-opt
./mlir/build_mlir.sh
# build onnx-mlir-opt
./onnx/build_onnx_mlir.sh
# build triton-opt
./triton/build.sh
# build circt-opt
./circt/build_circt.sh

2. Extract seed test cases from each subject's repositories:

cd /synthfuzz/eval
./mlir/find_seeds.sh
./onnx/find_seeds.sh
./triton/find_seeds.sh
./circt/find_seeds.sh

3. Optional only if you want to evaluate against NeuRI: For this step only, NeuRI needs to be run in its own container. Run the following outside the synthfuzz-artifact-icse2025 container:

cd synthfuzz-icse2025/eval/neuri
./start_docker.sh
./gen_indocker.sh  # inside the neuri-artifact container

Now returning to the synthfuzz-artifact-icse2025 container:

cd /synthfuzz/eval/neuri
python copy_models.py
python tf_to_onnx.py
python onnx_to_mlir.py
python onnx_to_onnx_mlir.py

3. Run each experiment:

# install computepairs
cd /synthfuzz/computepairs
go install

# ablation
cd /synthfuzz/eval/mlir/ablation/context && ./run.sh
cd /synthfuzz/eval/mlir/ablation && ./no_parameters.sh
cd /synthfuzz/eval/mlir/ablation && ./with_parameters.sh

# Coverage experiments

cd /synthfuzz/eval/mlirsmith && ./run.sh
cd /synthfuzz/eval/mlir/baseline && ./run.sh
cd /synthfuzz/eval/mlir/synthfuzz && ./run.sh
cd /synthfuzz/eval/mlir/grammarinator && ./run.sh
cd /synthfuzz/eval/mlir/mlirsmith && ./run.sh

cd /synthfuzz/eval/onnx/baseline && ./run.sh
cd /synthfuzz/eval/onnx/synthfuzz && ./run.sh
cd /synthfuzz/eval/onnx/grammarinator && ./run.sh
cd /synthfuzz/eval/onnx/mlirsmith && ./run.sh

cd /synthfuzz/eval/triton/baseline && ./run.sh
cd /synthfuzz/eval/triton/synthfuzz && ./run.sh
cd /synthfuzz/eval/triton/grammarinator && ./run.sh
cd /synthfuzz/eval/triton/mlirsmith && ./run.sh

cd /synthfuzz/eval/circt/baseline && ./run.sh
cd /synthfuzz/eval/circt/synthfuzz && ./run.sh
cd /synthfuzz/eval/circt/grammarinator && ./run.sh
cd /synthfuzz/eval/circt/mlirsmith && ./run.sh

# Only if step 3 was followed:
cd /synthfuzz/eval/mlir/neuri && ./run.sh
cd /synthfuzz/eval/onnx/neuri && ./run.sh
cd /synthfuzz/eval/triton/neuri && ./run.sh
cd /synthfuzz/eval/circt/neuri && ./run.sh