Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test reporting improvements #1039

Merged
merged 4 commits into from
Jul 26, 2019

Conversation

llongley
Copy link
Member

This implements a number of improvements to our test reporting, based on things both suggested to me and observed by me. Namely:

  • We now always report all test sub-results, even when a test failed all ten times - previously we'd just mark it as a single failure in that case and publish the first error message. We already had in place the ability to mark a test as failed if more than half of the sub-results failed, so we can simply always report tests tentatively as skipped and then update them later when processing test results.

  • Additionally, this adds the ability to generate a report on unreliable tests from CI builds X days before the present day that shows which tests were the most unreliable and which gives build links to see how they failed. This also changes things so we no longer mark a build as "succeeded with issues" when a test was unreliable - this is because we have enough unreliable tests that it's currently always being marked as such, such that that designation is losing meaning at the moment.

  • This also adds a new YAML build definition that outputs a report on the last 30 days every Sunday, which can be used to create issues for tests that are extremely unreliable. You can see an example of this in the "Output unreliable test report" step of this build:

https://dev.azure.com/ms/microsoft-ui-xaml/_build/results?buildId=24460

@llongley llongley requested review from jevansaks and kmahone July 12, 2019 05:01
@llongley llongley requested a review from a team as a code owner July 12, 2019 05:01
Copy link
Member

@kmahone kmahone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🕐

@llongley llongley merged commit 54dde88 into master Jul 26, 2019
@llongley llongley deleted the user/llongley/UnreliableTestReportingScript branch July 26, 2019 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants