Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fail harness if cover is not satisfied #2792

Open
camshaft opened this issue Sep 28, 2023 · 0 comments
Open

fail harness if cover is not satisfied #2792

camshaft opened this issue Sep 28, 2023 · 0 comments
Assignees
Labels
[C] Feature / Enhancement A new feature request or enhancement to an existing feature.

Comments

@camshaft
Copy link
Contributor

camshaft commented Sep 28, 2023

Requested feature: fail harness
Use case:

I just realized that the current bolero integration allows a proof to go through, even if you've specified a generator that is impossible to satisfy:

#[test]
#[kani::proof]
fn my_test() {
    bolero::check!().with_type::<u8>().filter(|_| false).for_each(|_| {
        panic!();
    });
}
$ cargo kani --tests --harness my_test --output-format=terse
Checking harness my_test...

VERIFICATION RESULT:
 ** 0 of 255 failed

 ** 0 of 1 cover properties satisfied (1 unreachable) // <= this should ideally fail the proof

VERIFICATION:- SUCCESSFUL
Verification Time: 0.14628594s

Complete - 1 successfully verified harnesses, 0 failures, 1 total.

Ideally, we'd replace the call to kani::assume with a cover call that would force harness failure. This could be called something like kani::cover_or_fail! or kani::assert_cover!.

Doing a test with multiple harnesses, it seems to behave as one would expect and only applies to the individual harness:

#[test]
#[kani::proof]
fn my_test() {
    bolero::check!().with_type::<u8>().for_each(|_| {
        // noop
    });
}

#[test]
#[kani::proof]
fn my_test() {
    bolero::check!().with_type::<u8>().filter(|_| false).for_each(|_| {
        panic!();
    });
}

#[kani::proof]
fn my_kani_proof() {
    assert_eq!(1 + 2, 3);
}
$ cargo kani --tests --output-format=terse
VERIFICATION:- SUCCESSFUL
Verification Time: 0.89504564s

Checking harness add_test...

VERIFICATION RESULT:
 ** 0 of 1502 failed (69 unreachable)

 ** 1 of 1 cover properties satisfied // <= the generator produced at least 1 valid value


VERIFICATION:- SUCCESSFUL
Verification Time: 0.8548021s

Checking harness my_test...

VERIFICATION RESULT:
 ** 0 of 255 failed

 ** 0 of 1 cover properties satisfied (1 unreachable) // <= the generator was UNSAT; should fail the harness!


VERIFICATION:- SUCCESSFUL
Verification Time: 0.14286228s

Checking harness my_kani_proof...

VERIFICATION RESULT:
 ** 0 of 2 failed  // <= this harness doesn't use bolero at all; no references to the cover property

VERIFICATION:- SUCCESSFUL
Verification Time: 0.024725173s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[C] Feature / Enhancement A new feature request or enhancement to an existing feature.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants