-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add "skipped" conditional for near zero metric value fast fails #412
Add "skipped" conditional for near zero metric value fast fails #412
Conversation
ENG-565 Servox fast-fail logic for handling values/thresholds at or near zero
During release candidate testing for 2.4 we ran into a failing baseline validation stage caused by the error_rate fast fail configuration:
The problem was that main error_rate was 0 whereas tuning error rate was 0.009090909090909090467524933388. The tuning error rate is within acceptable margins but because 0.009090909090909090467524933388 > 4 * 0 it fast-failed the measurement and we need to add logic to the fast fail to ignore values less than 1 or provide some sort of minimum value (eg. 0.25) when the threshold_metrics is detected as 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please also add a new property to the FastFailConfiguration class that toggles whether to treat absolute zero values (== 0, not close to 0) as missing since some metric systems will return 0 for a metric that has data in the system even though it has no data/null data for the time frame being queried. The new configuration allows us to account for that as needed
…rty to FastFailConfiguration for treating zeros as missing
…d pydantic prevalidator to slo_threshold_minimum
…pping per near-zero value
Co-authored-by: Fred L Sharp <[email protected]>
No description provided.