Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements a number of improvements to our test reporting, based on things both suggested to me and observed by me. Namely:
We now always report all test sub-results, even when a test failed all ten times - previously we'd just mark it as a single failure in that case and publish the first error message. We already had in place the ability to mark a test as failed if more than half of the sub-results failed, so we can simply always report tests tentatively as skipped and then update them later when processing test results.
Additionally, this adds the ability to generate a report on unreliable tests from CI builds X days before the present day that shows which tests were the most unreliable and which gives build links to see how they failed. This also changes things so we no longer mark a build as "succeeded with issues" when a test was unreliable - this is because we have enough unreliable tests that it's currently always being marked as such, such that that designation is losing meaning at the moment.
This also adds a new YAML build definition that outputs a report on the last 30 days every Sunday, which can be used to create issues for tests that are extremely unreliable. You can see an example of this in the "Output unreliable test report" step of this build:
https://dev.azure.com/ms/microsoft-ui-xaml/_build/results?buildId=24460
Our scripts now include parameters instead of strictly running based off of environment variables, which allows them to be run and debugged locally in addition to working in the lab.
Finally, this contains a few cleaning up of things, such as making it so we properly start at "Attempt Programmatically adding new NavigationViewItems to an existing NavigationView.MenuItems collection results in the added items rendering inside a generated NavigationViewItem node #1" when listing test sub-results, instead of starting at "Attempt #0".