Mock model for return_nan_when_prediction_changes tests

[aaarrti]

Description

• The tests for return_nan_when_prediction_changes parameter rely on high noise amplitudes, however if the predictions actually change is still not deterministic, and occasionally the tests will fail because of it.

Steps to reproduce the behavior

• Mock models outputs for tests with return_nan_when_prediction_changes=True

[aaarrti]

[annahedstroem]

Hi @aaarrti

What is meant by "however if the predictions actually change is still not deterministic, and occasionally the tests will fail because of it."? Do you mean that it is hard to predict if a model sometimes doesn't return the same predicted class after perturbation? Why will a test fail because of this?

Thank you!

[aaarrti]

Hi @aaarrti

What is meant by "however if the predictions actually change is still not deterministic, and occasionally the tests will fail because of it."? Do you mean that it is hard to predict if a model sometimes doesn't return the same predicted class after perturbation? Why will a test fail because of this?

Thank you!

It means, if this test was run, e.g., 100 times, it will fail once or twice

[aaarrti]

@annahedstroem here is an example of "unstable test" https://github.com/understandable-machine-intelligence-lab/Quantus/actions/runs/4519087543/jobs/7959357854?pr=240