Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing source-based coverage for a source file which is supposed to be covered #79456

Closed
marco-c opened this issue Nov 27, 2020 · 6 comments
Closed
Labels
A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) C-bug Category: This is a bug.

Comments

@marco-c
Copy link
Contributor

marco-c commented Nov 27, 2020

As I mentioned in rust-lang/compiler-team#278 (comment), I've implemented support for source-based code coverage in grcov, and started using it to collect coverage for grcov itself.

There is a file that is definitely covered, but doesn't appear to be in the report. It is in the report when using gcov-based coverage.

STR:

  1. Clone https://github.com/mozilla/grcov;
  2. Run LLVM_PROFILE_FILE="grcov-%p-%m.profraw" cargo test;
  3. Run grcov `find . \( -name "grcov-*.profraw" \) -print\` --binary-path ./target/debug/grcov -s . -t html --branch --ignore-not-existing --ignore "/*" -o coverage to generate a HTML report (or you can also generate a textual one using the llvm tools);
  4. See that src/llvm_tools.rs appears to be totally uncovered, while it should be covered.
@nagisa
Copy link
Member

nagisa commented Nov 27, 2020

Does this properly associate the profraw with the exact binary that produced it? --binary-path ./target/debug/grcov would suggest that you're using grcov (the [[bin]] target) while the profraw files seem to have been generated by various test binaries (typically have hashes in them, somewhere).

Is it also reproducible when using the llvm-cov tooling?

@camelid camelid added the A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) label Nov 27, 2020
@marco-c
Copy link
Contributor Author

marco-c commented Nov 27, 2020

Does this properly associate the profraw with the exact binary that produced it? --binary-path ./target/debug/grcov would suggest that you're using grcov (the [[bin]] target) while the profraw files seem to have been generated by various test binaries (typically have hashes in them, somewhere).

Oh, you're right, I was assuming only the source code behind #[cfg(test)] would be missed this way. It works if I use the actual test binary!
This means grcov will need a way to automatically detect the binary paths, and run llvm-cov accordingly...

@nagisa
Copy link
Member

nagisa commented Nov 27, 2020

Closing as not-a-bug (at least not in rustc).

Of course, I agree that having to associate the exact binary with the profile file it produced is sub-optimal, and cargo makes it harder still by appending hashes to everything. That said, this seems like the issue to be fixed in either the LLVM-provided instrumentation runtime (maybe just having a way for it to splice binary name into the profile filename would help?) or in cargo (that's what I'm planning to do for my own use-cases).

@nagisa nagisa closed this as completed Nov 27, 2020
@marco-c
Copy link
Contributor Author

marco-c commented Nov 27, 2020

That said, this seems like the issue to be fixed in either the LLVM-provided instrumentation runtime (maybe just having a way for it to splice binary name into the profile filename would help?) or in cargo (that's what I'm planning to do for my own use-cases).

Maybe using %m in LLVM_PROFILE_FILE, which "expands out to the instrumented binary’s signature". Then we just need to read the signature from the binaries to match them to the right profraw file.

@richkadel
Copy link
Contributor

@marco-c - In case this is helpful...

When PR #79109 lands, there will be an update to the "Rust Unstable Book" section on source-based-code-coverage, including an example for coverage of tests run through cargo test.

The example does use %m to make sure each test gets its own profraw result.

You can see the new version in the PR commit, here:

https://github.com/rust-lang/rust/blob/00117d712189ca703b481b471e11bcc742f0346e/src/doc/unstable-book/src/compiler-flags/source-based-code-coverage.md#test-coverage

Hope this helps.

@marco-c
Copy link
Contributor Author

marco-c commented Dec 2, 2020

@richkadel for now in grcov I'm simply running llvm-cov with the profdata file associated to all possible binary files, not a great solution, but it seems to work...

The next step will be to figure out how to use the binary signature to correctly match the profraw to the right binary, but I've looked at the binary signature code in LLVM and it is platform-specific, I'm not too keen on including something like that in grcov.
I've just now realized the binary signature is in the binary name itself! So that's going to make it easier ;)
The only constraint on the user will be the forced usage of LLVM_PROFILE_FILE.

For context, grcov is a tool to parse coverage artifacts (from Rust and other languages). Basically an alternative to llvm-cov, but easier to use and supporting more options, input formats, and output formats. I need to make it easy to use as part of CI, so can't rely on users running cargo test, checking the name of the binaries, and passing their names to grcov.

This is what I currently suggest doing to people using grcov:

LLVM_PROFILE_FILE="your_name-%p-%m.profraw" cargo test --verbose
grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "/*" -o lcov.info

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) C-bug Category: This is a bug.
Projects
None yet
Development

No branches or pull requests

4 participants