Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Report suppressed failures #687

Merged
merged 10 commits into from
Jul 21, 2023
Merged

feat: Report suppressed failures #687

merged 10 commits into from
Jul 21, 2023

Conversation

saig0
Copy link
Member

@saig0 saig0 commented Jul 17, 2023

Description

  • Create a new API to report suppressed failures
  • Mark the existing engine API as deprecated
  • Report a few suppressed failures, for example, for a missing variable. Other failures can be added later step-by-step.

Related issues

closes #260

@saig0 saig0 mentioned this pull request Jul 17, 2023
3 tasks
@saig0 saig0 force-pushed the 260-suppressed-failures branch 2 times, most recently from 5959e58 to 2503943 Compare July 17, 2023 09:17
@saig0 saig0 marked this pull request as ready for review July 17, 2023 09:17
@saig0 saig0 requested a review from remcowesterhoud July 17, 2023 09:18
Copy link
Contributor

@remcowesterhoud remcowesterhoud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @saig0! But much changes == much review comments 😄 Please have a look at them 👀

@saig0 saig0 requested a review from remcowesterhoud July 20, 2023 12:09
@saig0
Copy link
Member Author

saig0 commented Jul 20, 2023

@remcowesterhoud thank you for the detailed review. 🏅

I addressed all your comments. Please have another look. 🍰

Copy link
Contributor

@remcowesterhoud remcowesterhoud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes, all looks good to me 🏆

saig0 added 10 commits July 21, 2023 06:07
Adjust the Clirr check to ignore private generate functions. These functions contain a "$" symbol in their name.

Previously, the ignore rule was defined only for one function. Change the rule to include all generate functions.
Extend the evaluation context to collect evaluation failures. The collector is created with the context and is passed if a new context is created.

Allows to retrieve all evaluation failures by accessing the collector from the context.
Currently, the engine API returns an Either as the result of an evaluation. The Either contains the result value if the valuation was successful. Otherwise, it contains the failure.

To expose the suppressed failures as part of the result, we need a new API. We can't change the existing API because of the backward-compatibility.

Instead of duplicating the existing API in the engine class, the new API moved into a separate class and package. This should keep the API and the engine more clean.

To expose and promote the new API, we need a new builder that returns the new API. The existing builder returns the old API.
Extend the existing engine API to return an evaluation result.

Reorder the public API methods in the class and mark them as deprecated.

Adjust the root context to create a new context for each evaluation. This is required to collect the suppressed failures of the evaluation.
Create a new test class to verify that suppressed failures are reported for different cases.
Report suppressed failures in the interpreter (but for not all cases yet). The suppressed failures replace the previous logging statements.
Replace the logging in the built-in function by reporting these failures as suppressed failures.
The failure messages changed slightly. Adjust the expected failure messages in the existing test cases.
These functions are not used anymore.
Report a suppressed failure if a condition (e.g. conjunction, disjunction, if-then-else, etc.) is not a boolean value.
@saig0 saig0 force-pushed the 260-suppressed-failures branch from 2490737 to 435f3b5 Compare July 21, 2023 04:42
@saig0 saig0 merged commit 0d85ebb into main Jul 21, 2023
@saig0 saig0 deleted the 260-suppressed-failures branch July 21, 2023 04:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Customize the error handling
2 participants