WIP Code Coverage

[codemzs]

No description provided.

[TomFinley]

Thanks for working on this @codemzs . My understanding from @sharwell is that we can get the coverage test build from taking 10m during the test phase down to maybe 15m. But this depends upon future releases of Coverlet. (@sharwell correct me if I say anything wrong.) If we can get it down that far, might be worth taking Sam's advice elsewhere and putting it in main build.

I don't have full context on what is going on behind the scenes.

@glebuk and @shauheen (or indeed anyone), thoughts?

[sharwell]

There are two main ways to get the speed improved:

1. Threshold coverage to a single hit (Single hit coverlet-coverage/coverlet#309)
2. Targeted coverage exclusions using <Exclude>, starting with the following. These will be excluded for all test assemblies except the test assembly specifically intended to test the behavior of the type.
   • Microsoft.ML.Contracts
   • Microsoft.ML.Internal.Utilities
   • Microsoft.ML.Data.VBuffer<T>

@codemzs You could start by excluding the above three classes using <Exclude> in the Directory.Build.props files for all tests to observe the overall improvement, and then we can make that conditional once we confirm the improvement.

[codemzs]

@sharwell After excluding the classes you mentioned the build time with CC for Windows x64 debug went down to ~32 minutes from ~47 minutes. The build time without CC for the same platform is ~20 minutes. Link to build logs: https://dev.azure.com/dnceng/public/_build/results?buildId=75554

[sharwell]

@codemzs Those numbers should be good enough to hold us over until coverlet is updated 👍

[codemzs]

@sharwell Just to confirm, are you saying we should update the windows x64 debug leg of main CI build with the current coverlet nuget? If yes, I hope @TomFinley is cool too! :)

[sharwell]

@codemzs It could go in either leg. I'm saying this performance is enough that we can stop looking for <Exclude> items for now, and instead focus on getting the results uploaded to codecov.io.

[codemzs]

@sharwell Either leg as in? We are sticking with just one CI right?

[sharwell]

Meaning it can go in MachineLearning-CodeCoverage or in MachineLearning-CI. If it goes in MachineLearning-CI, we just remove MachineLearning-CodeCoverage.