List of tests to add a [Skip] attribute to

[vladimir-krestov]

This Issue is a storage of all new flaky tests that should be skipped.

• .NET Core Version: 7.0

Problem description:

• Some tests fail sometimes and block successful fixes deployment. We have to skip them now, to fix them later, but unblock CI build errors for successful PRs. The global Issue for fixing these tests: List of tests with [Skip] attribute #6690.

Proposed changes:

• Skip some test cases.
• Put skipped tests to #6690 to fix them later.

Expected behavior:

• Tests don't fail CI buiild for successful PRs

[vladimir-krestov]

@dreddy-work, @Tanya-Solyanik - please put new flaky tests here to skip them later.

[dreddy-work]

build # https://dev.azure.com/dnceng/public/_build/results?buildId=1620740&view=ms.vss-test-web.build-test-results-tab&runId=45061952&resultId=114518&paneView=debug

Two tests. I think these can be fixed instead of skipped but please check before skipping them.

[dreddy-work]

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1621721&view=ms.vss-test-web.build-test-results-tab&runId=45079044&resultId=115978&paneView=debug

[vladimir-krestov]

build # https://dev.azure.com/dnceng/public/_build/results?buildId=1620740&view=ms.vss-test-web.build-test-results-tab&runId=45061952&resultId=114518&paneView=debug

Two tests. I think these can be fixed instead of skipped but please check before skipping them.

@dreddy-work,

1. I have fixed DateTimePicker in Skipped more flaky tests #6731.
2. I couldn't fix the Control-ToolStrip test, because the bug doesn't reproduce to me, and investigation of code didn't show anything helpful.

[Tanya-Solyanik]

ARM64
https://dev.azure.com/dnceng/public/_build/results?buildId=1624825&view=ms.vss-test-web.build-test-results-tab

Error message
System.ArgumentException : dpiContext (Parameter '0')

Stack trace
at Interop.User32.SetThreadDpiAwarenessContext(IntPtr dpiContext) in //src/System.Windows.Forms.Primitives/src/Interop/User32/Interop.SetThreadDpiAwarenessContext.cs:line 25
at System.Windows.Forms.Tests.Dpi.ToolStripItemDpiTests.ToolStripItems_FontScaling(Int32 newDpi) in //src/System.Windows.Forms/tests/UnitTests/System/Windows/Forms/Dpi/ToolStripItemTests.Dpi.cs:line 54

[dreddy-work]

https://dev.azure.com/dnceng/public/_build/results?buildId=1627981&view=ms.vss-test-web.build-test-results-tab&runId=45199142&resultId=103869&paneView=debug

[RussKie]

This looks like a red herring, in the last week it failed twice https://runfo.azurewebsites.net/search/tests/?q=started%3A~7%20definition%3A267%20name%3A%22system.windows.forms.tests.datagridviewtests.datagridview_oncolumnheadersheightchanged_invokewithhandle_callscolumnheadersheightchanged%22

[RussKie]

@jaredpar has built a monitoring bot Runfo. This bot tracks builds and failures, we should monitor recurring failures, e.g.: failed tests in the last 7 days https://runfo.azurewebsites.net/search/tests/?q=started%3A%7E7+definition%3A267.
For tests that fail repeatedly we should create tracking issues:

Here's an example #6739 that we created yesterday, though we should consider making the subject a little more descriptive (e.g., DataGridView_RowHeadersWidth_SetWithHandle_GetReturnsExpected instead of setwithhandle_getreturnsexpected).

[dreddy-work]

Does it ignore fail after creating issue?

[RussKie]

No, the bot’s purpose is to track failures, not to modify the code. In general, I don’t think we should blindly skip every failing test, otherwise we may end up with all tests disabled.

[dreddy-work]

I don’t think we should blindly skip every failing test, otherwise we may end up with all tests disabled

Definitely not all. Only the tests that we think are failing because of possible interruptions to the events on lab machine or because of round errors while comparing the numeric values etc. In either case, intention is to confirm that is the case before disabling it.

[Tanya-Solyanik]

 System.Windows.Forms.Tests.DataGridViewTests.DataGridView_GridColor_Rendering
Source data is delayed, try again later
 Duration: 139 ms

Message: 
System.Windows.Forms.Metafiles.EmfValidator+WrappedXunitException :
Validator index 6: Polygon16Validator failed

[Tanya-Solyanik]

Intermittent failure:

https://dev.azure.com/dnceng/public/_build/results?buildId=1851676&view=ms.vss-test-web.build-test-results-tab&runId=48760536&resultId=103011&paneView=debug

System.Windows.Forms.Tests.Dpi.ToolStripItemDpiTests.ToolStripItems_FontScaling(newDpi: 192)
Error message
System.ArgumentException : dpiContext (Parameter '0')

Stack trace
at Interop.User32.SetThreadDpiAwarenessContext(IntPtr dpiContext) in //src/System.Windows.Forms.Primitives/src/Interop/User32/Interop.SetThreadDpiAwarenessContext.cs:line 25
at System.Windows.Forms.Tests.Dpi.ToolStripItemDpiTests.ToolStripItems_FontScaling(Int32 newDpi) in //src/System.Windows.Forms/tests/UnitTests/System/Windows/Forms/Dpi/ToolStripItemTests.Dpi.cs:line 54
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[RussKie]

Created #7365.

[Tanya-Solyanik]

Do we need this bug with the Runfo App aggregating failure statistics for us?

[merriemcgaw]

Given that we have the test-bug label that is added separately for each failure, we should just use that query to track the tests we need to fix. A separate tracking issue that aggregates them isn't needed.