Broken C++ Coverage with hermetic clang toolchain - No override for COVERAGE_GCOV_PATH and LLVM_COV possible with --test_env

[omar-droubi]

Description of the bug:

Hello Everyone,

Starting with Bazel 7.2.1 it's no longer possible to generate c++ code coverage using a hermetic Clang toolchain. I am using pw_toolchain_bazel to generate the toolchains definitions. Since our system doesn't have GCOV installed, and would like to use LLVM for the coverage, we override COVERAGE_GCOV_PATH and LLVM_COV with --test_env:

bazelrc content:

test --test_env=COVERAGE_GCOV_PATH=external/_main~_repo_rules~llvm_toolchain/bin/llvm-profdata
test --test_env=LLVM_COV=external/_main~_repo_rules~llvm_toolchain/bin/llvm-cov

By default Bazel extracts these two env variables from the tool_path for GCOV and LLVM_COV, but not from the action_configs for coverage actions.

cc_helper.bzl@1090

def _get_coverage_environment(ctx, cc_config, cc_toolchain):
    if not ctx.configuration.coverage_enabled:
        return {}

    env = {
        "COVERAGE_GCOV_PATH": cc_toolchain.tool_path(tool = "GCOV"),
        "LLVM_COV": cc_toolchain.tool_path(tool = "LLVM_COV"),
        "LLVM_PROFDATA": cc_toolchain.tool_path(tool = "LLVM_PROFDATA"),
        "GENERATE_LLVM_LCOV": "1" if cc_config.generate_llvm_lcov() else "0",
    }

Given that the current recommend way is to use action_configs and not tool_paths to define Action Tools, we do not set the tool path for GCOV and LLVM_COV, and their value is set to "".

This was fine since we could override their values using --test_env. However after the change 87b0a1f we can no longer override them.

This is happening because of the change in the order of env variables population in src/main/java/com/google/devtools/build/lib/exec/TestPolicy.java.

// Overwrite with the environment common to all tests, see --test_env.
// Omar: Here COVERAGE_GCOV_PATH and LLVM_COV take the correct values from --test_env
testAction.getConfiguration().getTestActionEnvironment().resolve(env, clientEnv);

// Rule-specified test env.
// Omar: getExtraTestEnv() has already COVERAGE_GCOV_PATH and LLVM_COV values defined 
// by cc_helper and they are set to "". These values overwrite what was supplied using --test_env
testAction.getExtraTestEnv().resolve(env, clientEnv);

Proposed Solutions:

1. _get_coverage_environment in cc_helper.bzl should use get_tool_for_action instead of tool_path.
2. Remove setting to default if tools are not found in _get_coverage_environment.
3. Explicit filtering for COVERAGE_GCOV_PATH, LLVM_COV, LLVM_PROFDATA,GENERATE_LLVM_LCOV in resolve function.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

WSL Ubuntu 20.04

What is the output of bazel info release?

release 7.2.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

87b0a1f

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@bazel-io flag

[fmeum]

@c-mita

[iancha1992]

@bazel-io fork 7.4.0

[omar-droubi]

Hi @c-mita / @fmeum : Is this topic still in the pipeline, or are you waiting for the full migration to rules_cc to work on it there ?

[c-mita]

Given that the current recommend way is to use action_configs and not tool_paths to define Action Tools, we do not set the tool path for GCOV and LLVM_COV, and their value is set to "".

I'm don't know if action_config does anything with the coverage tools - why not have the toolchain specify the path for the tools?

[omar-droubi]

In Pigweed it was not allowed to mix ActionConfig and plain tool_path, and also now in rules_cc, the toolchain config only accepts a cc_tool_map which is just a list of ActionConfigs. There is no way I can find to inject a plain old tool_path.

[keith]

i submitted #24670

[iancha1992]

A fix for this issue has been included in Bazel 7.5.0 RC2. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.5.0rc2. Thanks!

[axeluhlig]

We use --test_env=CC_CODE_COVERAGE_SCRIPT=external/cxx_tooling/cc_code_coverage_script.sh and still get the GCov does not exist at the given path: '' error when using Bazel 7.5.0rc2. It does not pick up the value we provide but instead uses external/bazel_tools/tools/test/collect_cc_coverage.sh.

I just tested it again, it Bazel 7.1 our value for CC_CODE_COVERAGE_SCRIPT gets propagated through.

Are we just using an unsupported feature here or is this bug only partially fixed?

[keith]

I think that one is complaining about either BAZEL_CC_COVERAGE_TOOL or GCOV, maybe setting one of those will sidestep?