Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update fork #13

Merged
merged 7 commits into from
Jan 30, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Documentation/Changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

-Fix ExcludeFromCodeCoverage attribute bugs [#129](https://github.com/tonerdo/coverlet/issues/129) and [#670](https://github.com/tonerdo/coverlet/issues/670) with [#671](https://github.com/tonerdo/coverlet/pull/671) by https://github.com/matteoerigozzi
-Fix bug with nested types filtering [#689](https://github.com/tonerdo/coverlet/issues/689)
-Fix Coverage Issue - New Using + Async/Await + ConfigureAwait [#669](https://github.com/tonerdo/coverlet/issues/669)

-Fix Coverage Issue - New Using + Async/Await + ConfigureAwait [#669](https://github.com/tonerdo/coverlet/issues/669)
-Improve branch detection for lambda functions and async/await statements [#702](https://github.com/tonerdo/coverlet/pull/702) by https://github.com/matteoerigozzi

### Improvements

Expand Down
91 changes: 62 additions & 29 deletions Documentation/VSTestIntegration.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,63 @@
# Coverlet Integration with VSTest
# Coverlet integration with VSTest (a.k.a. Visual Studio Test Platform)

As explained in quick start section to use collectors you need to run *SDK v2.2.401* or newer and your project file must reference `coverlet.collector.dll` and a minimum version of `Microsoft.NET.Test.Sdk`.
A sample project file looks like:

```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFrameworks>netcoreapp3.0;netcoreapp2.1;net46</TargetFrameworks>
</PropertyGroup>
<ItemGroup>
<!-- Temporary preview reference with essential vstest bug fix -->
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0-preview-20200116-01" />
<!-- Update this reference when new version is released -->
<PackageReference Include="coverlet.collector" Version="1.2.0">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
...
</ItemGroup>
...
</Project>
```
As you can see in sample above we're referencing a preview version of `Microsoft.NET.Test.Sdk`, this is temporary needed because there is a bug inside vstest platform during collectors loading([details](https://github.com/microsoft/vstest/issues/2205)). **At the moment there isn't a stable package released with fix so we need to use a preview**.

The reference to `coverlet.collector` package is included by default with xunit template test (`dotnet new xunit`), you only need to update the package for new versions like any other package reference.

| Entry point | How will code coverage be enabled? | Syntax |
|-------------|------------------------------------|----------------------------------------------------------------------|
|dotnet test CLI | Through a switch to condition data collection | `dotnet test --collect:"XPlat Code Coverage"` |
|dotnet vstest CLI | Through a switch to condition data collection | `dotnet vstest --collect:"XPlat Code Coverage"` |
With correct reference in place you can run coverage through default dotnet test CLI verbs:

NB. If you're using `dotnet vstest` you MUST `publish` your test project before i.e.
```bash
C:\project
```
dotnet test --collect:"XPlat Code Coverage"
```
or
```
dotnet publish
...
vstest -> C:\project\bin\Debug\netcoreapp3.0\testdll.dll
vstest -> C:\project\bin\Debug\netcoreapp3.0\publish\
... -> C:\project\bin\Debug\netcoreapp3.0\testdll.dll
... -> C:\project\bin\Debug\netcoreapp3.0\publish\
...
dotnet vstest C:\project\bin\Debug\netcoreapp3.0\publish\testdll.dll --collect:"XPlat Code Coverage"
```
As you can see in case of `vstest` verb you **must** publish project before.

At the end of tests you'll find the coverage file data under default vstest plat directory `TestResults`
```
Attachments:
C:\git\coverlet\Documentation\Examples\VSTest\HelloWorld\XUnitTestProject1\TestResults\bc5e983b-d7a8-4f17-8c0a-8a8831a4a891\coverage.cobertura.xml
Test Run Successful.
Total tests: 1
Passed: 1
Total time: 2,5451 Seconds
```
You can change the position of files using standard `dotnet test` switch `[-r|--results-directory]`
*NB: By design vstest platform will create your file under a random named folder(guid string) so if you need stable path to load file to some gui report system(i.e. coveralls, codecov, reportgenerator etc..) that doesn't support glob patterns or hierarchical search, you'll need to manually move resulting file to a predictable folder*

## Coverlet options supported by VSTest integration

At the moment VSTest integration doesn't support all features of msbuild and .NET tool, for instance show result on console, report merging and threshold validation.
We're working to fill the gaps.

### Coverlet Options Supported with VSTest

#### Default
| Option | Summary |
Expand Down Expand Up @@ -61,32 +101,25 @@ How to specify these options via runsettings?
</DataCollectionRunSettings>
</RunSettings>
```
Filtering details are present on [msbuild guide](https://github.com/tonerdo/coverlet/blob/master/Documentation/MSBuildIntegration.md#excluding-from-coverage).

This runsettings file can easily be provided using command line option as given :

1. `dotnet test --settings coverlet.runsettings`
1. `dotnet test --collect:"XPlat Code Coverage" --settings coverlet.runsettings`

2. `dotnet vstest --settings coverlet.runsettings`
2. `dotnet vstest C:\project\bin\Debug\netcoreapp3.0\publish\testdll.dll --collect:"XPlat Code Coverage" --settings coverlet.runsettings`

Take a look at our [`HelloWorld`](Examples/VSTest/HelloWorld/HowTo.md) sample.

## Implementation Details
## How it works

The proposed solution is implemented with the help of [datacollectors](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md).
Coverlet integration is implemented with the help of [datacollectors](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md).
When we specify `--collect:"XPlat Code Coverage"` vstest platform tries to load coverlet collectors present inside `coverlet.collector.dll`

1. Outproc Datacollector : The outproc collector would always run in a separate process(datacollector.exe/datacollector.dll) than the process in which tests are being executed(testhost*.exe/testhost.dll). This datacollector would be responsible for calling into coverlet APIs for instrumenting dlls, collecting coverage results and sending the coverage output file back to test platform.
1. Outproc Datacollector : The outproc collector run in a separate process(datacollector.exe/datacollector.dll) than the process in which tests are being executed(testhost*.exe/testhost.dll). This datacollector is responsible for calling into coverlet APIs for instrumenting dlls, collecting coverage results and sending the coverage output file back to test platform.

2. Inproc Datacollector : The inproc collector in the testhost process executing the tests. This collector will be needed to remove the dependency on the exit handler to flush the hit files.

The datacollectors will be bundled as a separate NuGet package, the reference to which will be added by default in the .NET Core test templates, thus making it the default solution for collecting code coverage for .NET core projects.
```
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="x.x.x" />
<PackageReference Include="MSTest.TestAdapter" Version="x.x.x" />
<PackageReference Include="MSTest.TestFramework" Version="x.x.x" />
<PackageReference Include="coverlet.collector" Version="x.x.x" />
</ItemGroup>
```
2. Inproc Datacollector : The inproc collector is loaded in the testhost process executing the tests. This collector will be needed to remove the dependency on the process exit handler to flush the hit files and avoid to hit this [serious know issue](https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test)

## Know issue
## Known Issues

Thre is a know issue, check it here https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#2-upgrade-coverletcollector-to-version--100
For a comprehensive list of known issues check the detailed documentation https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Coverlet is a cross platform code coverage framework for .NET, with support for
# Main contents
* [QuickStart](#Quick-Start)
* [How It Works](#How-It-Works)
* [Know Issues](#Know-Issues)
* [Known Issues](#Know-Issues)
* [Consume nightly build](#Consume-nightly-build)
* [Feature samples](Documentation/Examples.md)
* [Cake Add-In](#Cake.-Add-In)
Expand Down Expand Up @@ -125,7 +125,7 @@ _Note: The assembly you'd like to get coverage for must be different from the as

## Are you in trouble with some feature? Check on [examples](Documentation/Examples.md)!

## Know Issues
## Known Issues

Unfortunately we have some know issues, check it [here](Documentation/KnowIssues.md)

Expand Down
3 changes: 3 additions & 0 deletions eng/azure-pipelines-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ steps:
- task: UseDotNet@2
inputs:
version: 2.2.402
- task: UseDotNet@2
inputs:
version: 3.1.100
- powershell:
.\eng\nightly.ps1 -apiKey $env:APIKEY -source $env:SOURCE
ignoreLASTEXITCODE: true
Expand Down
35 changes: 25 additions & 10 deletions src/coverlet.core/Symbols/CecilSymbolHelper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,26 @@ internal static class CecilSymbolHelper
{
private const int StepOverLineCode = 0xFEEFEE;

private static bool IsMoveNextInsideAsyncStateMachine(MethodDefinition methodDefinition)
// In case of nested compiler generated classes, only the root one presents the CompilerGenerated attribute.
// So let's search up to the outermost declaring type to find the attribute
private static bool IsCompilerGenerated(MethodDefinition methodDefinition)
{
if (!methodDefinition.FullName.EndsWith("::MoveNext()"))
TypeDefinition declaringType = methodDefinition.DeclaringType;
while (declaringType != null)
{
return false;
if (declaringType.CustomAttributes.Any(ca => ca.AttributeType.FullName == typeof(CompilerGeneratedAttribute).FullName))
{
return true;
}
declaringType = declaringType.DeclaringType;
}

if (methodDefinition.DeclaringType.CustomAttributes.Count(ca => ca.AttributeType.FullName == typeof(CompilerGeneratedAttribute).FullName) > 0)
return false;
}

private static bool IsMoveNextInsideAsyncStateMachine(MethodDefinition methodDefinition)
{
if (methodDefinition.FullName.EndsWith("::MoveNext()") && IsCompilerGenerated(methodDefinition))
{
foreach (InterfaceImplementation implementedInterface in methodDefinition.DeclaringType.Interfaces)
{
Expand Down Expand Up @@ -72,7 +84,8 @@ private static bool IsRecognizedMoveNextInsideAsyncStateMachineProlog(MethodDefi
methodDefinition.Body.Instructions[0].OpCode == OpCodes.Ldarg) &&

methodDefinition.Body.Instructions[1].OpCode == OpCodes.Ldfld &&
methodDefinition.Body.Instructions[1].Operand is FieldDefinition fd && fd.Name == "<>1__state" &&
((methodDefinition.Body.Instructions[1].Operand is FieldDefinition fd && fd.Name == "<>1__state") ||
(methodDefinition.Body.Instructions[1].Operand is FieldReference fr && fr.Name == "<>1__state")) &&

(methodDefinition.Body.Instructions[2].OpCode == OpCodes.Stloc &&
methodDefinition.Body.Instructions[2].Operand is VariableDefinition vd && vd.Index == 0) ||
Expand Down Expand Up @@ -104,7 +117,7 @@ public static List<BranchPoint> GetBranchPoints(MethodDefinition methodDefinitio

/*
If method is a generated MoveNext we'll skip first branches (could be a switch or a series of branches)
that check state machine value to jump to correct state(for instance after a true async call)
that check state machine value to jump to correct state (for instance after a true async call)
Check if it's a Cond_Branch on state machine current value int num = <>1__state;
We are on branch OpCode so we need to go back by max 2 operation to reach ldloc.0 the load of "num"
Max 2 because we handle following patterns
Expand Down Expand Up @@ -148,7 +161,7 @@ More tha one branch

so we know that current branch are checking that field and we're not interested in.
*/
if (isAsyncStateMachineMoveNext && isRecognizedMoveNextInsideAsyncStateMachineProlog)
if (isRecognizedMoveNextInsideAsyncStateMachineProlog)
{
bool skipInstruction = false;
Instruction current = instruction.Previous;
Expand All @@ -171,16 +184,18 @@ so we know that current branch are checking that field and we're not interested

// Skip get_IsCompleted to avoid unuseful branch due to async/await state machine
if (
isAsyncStateMachineMoveNext && instruction.Previous.Operand is MethodReference operand &&
isRecognizedMoveNextInsideAsyncStateMachineProlog && instruction.Previous.Operand is MethodReference operand &&
operand.Name == "get_IsCompleted" &&
(
operand.DeclaringType.FullName.StartsWith("System.Runtime.CompilerServices.TaskAwaiter") ||
operand.DeclaringType.FullName.StartsWith("System.Runtime.CompilerServices.ConfiguredTaskAwaitable")
operand.DeclaringType.FullName.StartsWith("System.Runtime.CompilerServices.ConfiguredTaskAwaitable") ||
operand.DeclaringType.FullName.StartsWith("System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable")
)
&&
(
operand.DeclaringType.Scope.Name == "System.Runtime" ||
operand.DeclaringType.Scope.Name == "netstandard"
operand.DeclaringType.Scope.Name == "netstandard" ||
operand.DeclaringType.Scope.Name == "System.Threading.Tasks.Extensions"
)
)
{
Expand Down
12 changes: 5 additions & 7 deletions test/coverlet.core.tests/Coverage/CoverageTests.Lambda.cs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ public void Lambda_Issue343()
{
instance.InvokeAnonymous_Test();
((Task<bool>)instance.InvokeAnonymousAsync_Test()).ConfigureAwait(false).GetAwaiter().GetResult();

return Task.CompletedTask;
}, persistPrepareResultToFile: pathSerialize);
return 0;
Expand All @@ -34,15 +35,12 @@ public void Lambda_Issue343()
// Expected branches
(22, 0, 0),
(22, 1, 1),
(50, 2, 0),
(50, 3, 1),
// Unexpected branches
(50, 0, 0),
(50, 1, 1),

// Unexpected branches - generated by compiler to cache delegate instance
(20, 0, 1),
(20, 1, 1),
(49, 0, 1),
(49, 1, 0),
(54, 4, 0),
(54, 5, 1),
(48, 0, 1),
(48, 1, 1)
);
Expand Down
3 changes: 2 additions & 1 deletion test/coverlet.integration.tests/Msbuild.cs
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,8 @@ public void Test_MultipleTargetFrameworkReport_CoverletOutput_Folder()

foreach (string targetFramework in targetFrameworks)
{
Assert.True(File.Exists(Path.Combine(clonedTemplateProject.ProjectRootPath, $"coverage.{targetFramework}.json")));
string fileToCheck = Path.Combine(clonedTemplateProject.ProjectRootPath, $"coverage.{targetFramework}.json");
Assert.True(File.Exists(fileToCheck), $"Expected file '{fileToCheck}'\nOutput:\n{standardOutput}");
}

AssertCoverage(clonedTemplateProject, "coverage.*.json");
Expand Down