Rustc is known to report an incorrect coverage for some lines https://stackoverflow.com/questions/32521800/why-does-kcov-calculate-incorrect-code-coverage-statistics-for-rust-programs.
rust-covfix
will read coverage from the file generated by grcov, fix it, then outputs the correct coverage.
Though only lcov
format is supported at current, Another formats are going to be supported in future releases.
- Compatible with the latest stable/beta/nightly Rust compiler
- Windows/OSX/Linux are all supported
- Lightweight (small dependencies)
- Fast and safe (implemented in Rust language)
rust-covfix
is also available with Rust API (Documentation)- Show summary of coverage difference.
Optional features are available with cargo's --features
option. You can specify the features like:
$ cargo install --no-default-features --features "cli lcov"
Feature name | Description | Default? |
---|---|---|
cli | Command Line Interface. This feature is required to build rust-covfix executable. |
yes |
lcov | Make LcovParser available | yes |
noinline | Avoid adding #cfg[inline] attribute on function. (deprecated) |
no |
backtrace | Dump backtrace information on every time the error has occured. | no |
Download the latest release from GitHub Release Page.
You can also install via cargo
command.
$ cargo install rust-covfix
It seems that the current version of rustc (1.42) will automatically inline the function that is only called from one place. This behaviour causes incorrect coverage for your tests.
To avoid this, you have to add #[inline(never)]
attributes for their functions manually. I recommend defining a new feature flag coverage
in your crate. In Cargo.toml, append the following lines.
[features]
coverage = []
And then, add the attribute like #[cfg(feature = "coverage", inline(never))]
to the functions which is called from just one location.
#[cfg(feature = "coverage", inline(never))]
fn foo() {
// ...
}
This will avoid inlining the functions only when you enable coverage
feature flag.
In order to generate code coverage with rustc, you must specify -Zprofile
option.
This option is currently (1.42) unstable and only available from nightly toolchain.
Also, some other flags will be required to generate correct coverage. Here is my recommend.
$ export CARGO_INCREMENTAL=0
$ export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort"
Then, compile your crate and run tests.
$ cargo test --features coverage
Now there are code coverage data in target/debug/deps
directory.
The next step is aggregating them and convert the format so that rust-covfix
can read coverage data.
I highly recommend to use grcov to aggregate them. This project is developed by mozilla team and supports the latest Rust toolchains.
Install latest grcov
and run the following commands from your project root directory.
zip -0 ccov.zip `find . \( -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" \) -print`
./grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov.info
where YOUR_PROJECT_NAME
is your crate name specified in Cargo.toml
.
Now rust-covfix can read coverage from lcov.info
.
$ rust-covfix -o lcov_correct.info lcov.info
This command will write a correct coverage into lcov_correct.info
. You can upload them into codecov.io, or generate HTML summary using genhtml
.
Here is an example script to use rust-covfix
on Travis CI environment.
#!/bin/bash
# ci/script.sh
set -ex
if [ "$TRAVIS_RUST_VERSION" = "nightly" ] && [ -z "$TRAVIS_TAG" ]; then
# Setup grcov
wget https://github.com/mozilla/grcov/releases/download/v0.5.7/grcov-linux-x86_64.tar.bz2
tar xvf grcov-linux-x86_64.tar.bz2
# Setup environmental variables
export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort"
fi
# Compile and run tests
cargo test --all-features
if [ "$TRAVIS_RUST_VERSION" = "nightly" ] && [ -z "$TRAVIS_TAG" ]; then
# collect coverages
zip -0 ccov.zip `find . \( -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" \) -print`
./grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov.info
# fix coverage using rust-covfix
rust-covfix lcov.info -o lcov.info
# upload coverage to codecov
bash <(curl -s https://codecov.io/bash) -f lcov.info
fi
Then, call this script from travis.yml
language: rust
matrix:
include:
- os: linux
rust: nightly
script:
- bash ci/script.sh
codecov:
stage: coverage
artifacts:
name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
paths:
- artifacts/
variables:
# Variables partly came from https://github.com/mozilla/grcov/blob/master/README.md
CARGO_INCREMENTAL: 0
RUSTFLAGS: "-Zprofile -Zmir-opt-level=0 -Ccodegen-units=1
-Copt-level=0 -Clink-dead-code -Coverflow-checks=off"
# We removed the `-Cinline-threshold=0` flag from the above `RUSTFLAGS` since it was bugged
# at the time and lead to inlining of functions that shouldn't be inlined for the coverage
# report. (More information here: https://github.com/Kogia-sima/rust-covfix/issues/2)
script:
# RUSTFLAGS are the cause target cache can't be used here
- unset "CARGO_TARGET_DIR" # only needed if it's not default
- cargo clean
- cargo test --verbose --all-features --no-fail-fast --workspace
- cargo build --verbose --all-features --workspace
# coverage with branches
- grcov ./target -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov-w-branch.info
- rust-covfix -o lcov-w-branch_correct.info lcov-w-branch.info
# We'd like to not use a remote bash script for uploading the coverage reports,
# however this job seems to be more tricky than we hoped.
- bash <(curl -s https://codecov.io/bash) -t "$CODECOV_P_TOKEN" -f lcov-w-branch_correct.info
- cp *.info /artifacts # this is to output the report files, for the debug purposes
name: Coverage
on: [push, pull_request]
jobs:
coverage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Install toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
target: x86_64-unknown-linux-gnu
toolchain: nightly
override: true
- name: Install grcov
run: curl -L https://github.com/mozilla/grcov/releases/download/v0.6.1/grcov-linux-x86_64.tar.bz2 | tar jxf -
- name: Install rust-covfix
run: |
curl -L https://github.com/Kogia-sima/rust-covfix/releases/download/v0.2.2/rust-covfix-linux-x86_64.tar.xz |tar Jxf -
mv rust-covfix-linux-x86_64/rust-covfix ./
- name: Test all crates
env:
CARGO_INCREMENTAL: 0
RUSTFLAGS: -Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort
RUSTDOCFLAGS: -C panic=abort
run: |
cargo build --all-features --workspace
cargo test --all-features --workspace
- name: collect coverages
run: |
zip -0 ccov.zip `find . \( -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" \) -print`
./grcov ccov.zip --llvm --branch -t lcov -o lcov.info --ignore "/*" --ignore "sailfish-tests/*"
- name: fix coverages
run: ./rust-covfix -o lcov.info lcov.info
- name: upload coverage
uses: codecov/codecov-action@v1
with:
file: ./lcov.info
There are 3 reasons for this.
-
A grcov collaborator is aware of rustc reporting incorrect coverage mozilla/grcov#249 (comment). I'm trying to compare original Rust code and transformed LLVM IR, but still don't fully understand why Rustc report incorrect coverage. The
rust-covfix
was originally developed as a temporal workaround for this problem until I understand when and where rustc reports incorrect coverage. It is not completely unravelled yet. -
grcov
is just a coverage collection tool, not a coverage generator. It only aggregates the raw coverage data, and convert the format. It does not read the source code, nor manipulate any coverage data. Also, although grcov is targetting many programming languages such as C/C++, Nim and Rust,rust-covfix
only supports Rust. -
rust-covfix
is targetting not only grcov, but also cargo-kcov and Tarpaulin in the future. Actually,Tarpaulin
can now generate thelcov.info
. It meansrust-covfix
is already able to fix the coverage data generated byTarpaulin
. I'm working on feature/cobertura branch to supportcargo-kcov
too. The final objective is to support these 3 tools.
rust_covfix
fixes the coverage information using some rules. You can pass --rules
option to specify which rules are used to fix coverages.
closing brackets, line of else
block will be ignored.
if a > 0 {
b = a
} else { // <-- marked as "not executable"
b = -a
}; // <-- marked as "not executable"
module block named test
or tests
which has attribute cfg(test)
will be ignored.
all functions with #[test]
attribute will be also ignored.
#[cfg(test)]
mod tests { // <-- removed from coverage
fn util() { ... } // <-- removed from coverage
#[test]
fn test_hoge() { ... } // <-- removed from coverage
}
Fix rust internal bugs that loop branches are not correctly passed.
for i in 0..10 { // <-- fix branch coverage information
println!("{}", i);
}
structs with derive(...)
attribute will be ignored
#[derive(Clone, Debug)] // <-- removed from coverage
struct Point { // <-- removed from coverage
x: f64, // <-- removed from coverage
y: f64 // <-- removed from coverage
} // <-- removed from coverage
ignore coverage based on comment marker
fn main() {
let a = 1 + 2; // cov:ignore-line
// cov:begin-ignore-branch
println!("Hello");
println!("world!");
// cov:end-ignore-branch
// cov: begin-ignore-line
if a > 2 {
println!("a is large!");
} else if a == 0 {
println!("a is small!");
}
// cov: end-ignore-line
// cov:begin-ignore
println!("a = {}", a);
// cov:end-ignore
println!("finish."); // cov:ignore-branch
return (); // cov:ignore
}
- Support
cobertura.xml
file. (WIP) - Add option for uploading the correct coverages to coveralls.
- Use syntax tree generated using syn crate.
- Performance improvement
π€ Kogia-sima
- Twitter: @Kogia_sima
- Github: @Kogia-sima
Contributions, issues and feature requests are welcome!
Feel free to check issues page.
Give a βοΈ if this project helped you!
Copyright Β© 2019 Kogia-sima.
This project is MIT licensed.
This README was generated with β€οΈ by readme-md-generator