Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lightweight .NET Standard version of the Engine #205

Merged
merged 30 commits into from
Apr 28, 2017
Merged

Lightweight .NET Standard version of the Engine #205

merged 30 commits into from
Apr 28, 2017

Conversation

rprouse
Copy link
Member

@rprouse rprouse commented Mar 30, 2017

I would like to start reviewing this approach, but we should not merge until end-to end testing with the adapter has been performed. I will add some comments inline with the code as to some of the decisions that I made, but the overall approach is,

We need versions of the VS Adapter that targets the full CLR, .NET Core and UWP to run tests for those platforms in Visual Studio and with the newer version of the dotnet test command. Our adapter uses the engine in-process and without many of the extensions, so that is what I am targeting. At first, I attempted to use the nunit.portable.agent project, but the majority of that code was unused and I was pulling in too much code from this project, so I switched back to creating a .NET Standard version of the engine.

  • This version of the engine is only intended to be used in-process. If we support running .NET Core tests from our current desktop console runner, it will launch a .NET Core agent that uses this engine, but the console runner will not.
  • I decided to make it a different NuGet package to differentiate the purpose
  • This does not support extensions. There is very little support in .NET Standard for loading assemblies from other directories which makes the plugin model difficult.
  • There is no support for AppDomains or separate processes
  • This only supports classes needed for the Visual Studio adapter. We can add more as needed, but I think we should keep it lean

For now, I plan on using CI builds of the NuGet package in the adapter, then release an alpha version of this and the adapter so we can get people in the community who are using .NET Core to test and give feedback.

Fixes #10

@@ -49,7 +51,14 @@ public TestPackage(string filePath)

if (filePath != null)
{
#if NETSTANDARD1_3
if (!Path.IsPathRooted(filePath))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one major difference between the full engine and this version. The paths in .NET Standard vary dramatically depending on how you run the project. For example, it you use dotnet run, it compiles a project and runs it. The assembly is in a sub-directory, but the current directory is the location of the project file. If you use dotnet execute the path is that of the assembly. Because of this, our code that figures out the full path does not work in all cases, so I decided to only allow full paths and leave it up to the calling runner to resolve that path.

/// DirectTestRunner is the abstract base for runners
/// that deal directly with a framework driver.
/// </summary>
public class DirectTestRunner : AbstractTestRunner
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any engine files that were dramatically different I created a new copy for the .NET Standard engine. They were getting way too complicated with the #if statements.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the parts in common could be abstracted to a base class as well, at some point, to help manage the complexity? Things like NUnitNetStandardDriverTests could then be consolidated and test all the drivers at once via a common interface at least.


[Test]
[Test]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we have some old mixed tabs and spaces in this file.

@@ -86,7 +86,8 @@ public void Canonicalize()
PathUtils.Canonicalize( @"folder1\folder2\..\..\..\file.tmp" ) );
}

[Test]
#if !NETCOREAPP1_1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are all a .NET Core app because they are self-executing NUnitLite.


foreach (XmlElement element in doc.DocumentElement["Settings"].ChildNodes)
{
if (element.Name != "Setting")
throw new ApplicationException("Unknown element in settings file: " + element.Name);
throw new Exception("Unknown element in settings file: " + element.Name);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ApplicationException is not available 😦

#if NETSTANDARD1_3
default:
return null;
#else
case "user":
return new XmlTransformResultWriter(args);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None of the XML transform classes we use are available and they aren't used in the VS Adapter.

build.ps1 Outdated
@@ -89,7 +89,7 @@ if(!$PSScriptRoot){
$TOOLS_DIR = Join-Path $PSScriptRoot "tools"
$NUGET_EXE = Join-Path $TOOLS_DIR "nuget.exe"
$CAKE_EXE = Join-Path $TOOLS_DIR "Cake/Cake.exe"
$NUGET_URL = "https://dist.nuget.org/win-x86-commandline/latest/nuget.exe"
$NUGET_URL = "https://dist.nuget.org/win-x86-commandline/v4.0.0/nuget.exe"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The .NET Core restore needs the 4.0 version of NuGet, but latest is currently pointing at 3.5. Once latest is updated, this can be reverted.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran into this as well. An alternative which I would recommend is to leave the bootstrapper alone and replace nuget restore with msbuild /t:restore.
See https://gitter.im/cake-build/cake?at=58d156017b3f37e7541e55b0 and http://blog.nuget.org/20170316/NuGet-now-fully-integrated-into-MSBuild.html

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching to MSBuild would require v15 which would force all team members and build machines to use Visual Studio 2017.

@rprouse
Copy link
Member Author

rprouse commented Mar 30, 2017

I can't make much of the build failures on Travis. If anyone has ideas, I would appreciate the help, otherwise I may just not build .NET Core on Linux/Mac 😄

Could be that Travis hasn't updated .NET core, but it is showing as not installed, so it shouldn't be building and shouldn't be giving MSBuild errors. Possibly because I am using the newer DotNetCore cake tasks? Maybe we need to run --experimental on Travis?

@ghost
Copy link

ghost commented Mar 31, 2017

@rprouse - is this any help?

https://github.com/dotnet/docs/blob/master/docs/core/tools/dotnet-install-script.md

You can install the runtime by way of a bash script.

@rprouse
Copy link
Member Author

rprouse commented Mar 31, 2017

@Fir3pho3nixx thanks for that, it might be helpful.

@ghost
Copy link

ghost commented Apr 1, 2017

Figured this out, the main battle here is figuring out how to get libunwind working on travis using ubuntu trusty.

This is my travis yaml file that worked.

language: csharp

solution: fortress.sln

mono: none

dotnet: none

sudo: required

dist: trusty

before_install:  

    - sudo apt-get -qq update
    
    - sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
  
    - sudo apt-get install -y libunwind8

    - wget -O - https://raw.githubusercontent.com/dotnet/cli/rel/1.0.0/scripts/obtain/dotnet-install.sh | bash -s -- -i ./dotnetcli/

script:

    - ./dotnetcli/dotnet restore ./src/Fortress.Windsor/Fortress.Windsor.csproj
    
    - ./dotnetcli/dotnet restore ./src/Fortress.Facilities.Logging/Fortress.Facilities.Logging.csproj
    
    - ./dotnetcli/dotnet restore ./src/Fortress.Windsor.Tests/Fortress.Windsor.Tests.csproj
    
    - ./dotnetcli/dotnet test ./src/Fortress.Windsor.Tests/Fortress.Windsor.Tests.csproj

@rprouse
Copy link
Member Author

rprouse commented Apr 19, 2017

I am going to disable building .NET Core on Travis, but in the meantime, this is ready for review. It creates a .NET Standard version of the engine with the limitations mentioned above. It is working well in the Visual Studio adapter to run tests in Visual Studio and on the command line with vstest and dotnet test.

My thinking is to release an alpha version of the .NET Standard engine to NuGet and consume it in an alpha release of the adapter.

@rprouse
Copy link
Member Author

rprouse commented Apr 19, 2017

@nunit/engine-team I cannot seem to get Travis building for the life of me. It is failing in the NuGet restore of the main project for some Travis builds. The nuget restore hasn't changed and the main solution hasn't changed. The only difference that I can think of is that there are project.json files in some of the projects. That is no different than the nunit framework repo though.

I turned up the verbosity of the nuget restore and it is showing the error

NuGet.CommandLine.CommandLineException: 
  at NuGet.CommandLine.MsBuildUtility+<GetProjectReferencesAsync>d__5.MoveNext () <0x40fa8830 + 0x0125b> in <filename unknown>:0 

The successful Travis build is finding MSBuild 15, but this is finding MSBuild 14. My Windows box finds 14 and succeeds though.

Any help or ideas would be appreciated.

@rprouse
Copy link
Member Author

rprouse commented Apr 22, 2017

I've managed to get this building on my Linux box. At this point, it doesn't look like it will require any code changes or changes to build.cake, it will only be changes to the Travis environment to ensure the correct tools are installed on Linux and Mac.

Based on that, this is ready to review. I would like feedback from the @nunit/core-team and @nunit/engine-team on the approach. I would also like to do an alpha release of just the .NET Core engine soon.

@CharliePoole, you also had concerns about calling this an engine. I would appreciate it if you could take a look and see if you still feel that way or possibly suggest better naming to prevent confusion.

One other major question is the fact that I copied and rewrote several files rather than use too many #if blocks. I worry that decision might cause maintenance problems in the future. Most of the files are shared between the two solutions, but the following files are copies,

  • TestEngine.cs
  • DirectTestRunner.cs
  • MasterTestRunner.cs
  • DefaultTestRunnerFactory.cs
  • DriverService.cs

@rprouse
Copy link
Member Author

rprouse commented Apr 25, 2017

CI is now passing on this PR. The Linux Mono 4.8.1 failure is unrelated, see #211.

This can be reviewed and merged when ready.

Copy link
Member

@ChrisMaddock ChrisMaddock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! It's a lot of code to review as one, but there's nothing glaringly obvious to me which seems out of place.

I agree the duplication is a concern - but I'd personally prioritise getting an alpha version out there with a view to deal with this later. What's the long term plan - I presume it's not a separate mini-engine?


// TODO: What about bad extensions?
return new DirectTestRunner(package);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method seems a little odd now with all the project stuff stripped out. Return an AggregatingTestRunner if there are 'any' assemblies, and a DirectTestRunner only if there are zero? If extensions aren't supported, shouldn't everything be an assembly, and anything else invalid?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, but I didn't want to strip out too much that we might end up re-adding. We don't need NUnit project loading or solution loading for the VS Adapter, but I thought that once we move to other runners like Xamarin or UWP, we might end up adding some of the extensions back in.

}

[Test]
public void AssemblyPathIsUsedAsFilePath()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be AssemblyPathIsUsedAsFullName?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, fixing.

@rprouse
Copy link
Member Author

rprouse commented Apr 28, 2017

@ChrisMaddock thanks for the review, I have pushed an update for one of your comments.

@jnm2 I would appreciate a second code review on this before I merge if you have some time.

@nunit/core-team I plan on putting out an alpha1 release of this once it is merged so that it can be put into the adapter and we can get an alpha1 release of that out too. I was planning on calling this 3.7.0-alpha1 in prep for the final 3.7 release. I was planning on calling the adapter 4.0.0-alpha1 because it is a major change in functionality.

Does anyone see any problem with that plan or does anyone have suggestions?

Copy link
Collaborator

@jnm2 jnm2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an impressive amount of work and I can't claim to completely understand it, but I can't wait to have an excuse to try it out!

<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
<GenerateAssemblyProductAttribute>false</GenerateAssemblyProductAttribute>
<GenerateAssemblyTitleAttribute>false</GenerateAssemblyTitleAttribute>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These 9 lines can be replaced with <GenerateAssemblyInfo>false</GenerateAssemblyInfo>

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That cleans things up, thanks. Do you know where all of the new options are documented? I've just been looking through examples of Microsoft projects.

</PropertyGroup>
<PropertyGroup>
<AssemblyOriginatorKeyFile>..\..\nunit.snk</AssemblyOriginatorKeyFile>
<GeneratePackageOnBuild>False</GeneratePackageOnBuild>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Off the top of my head I think this is the default and this line could be left out?

<RootNamespace>NUnit.Tests</RootNamespace>
<TargetFramework>netstandard1.6</TargetFramework>
<AssemblyName>mock-assembly</AssemblyName>
<IntermediateOutputPath>obj\$(Configuration)\netstandard\</IntermediateOutputPath>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure this is the default and this line could be left out?

@@ -39,7 +40,7 @@ public interface IFrameworkDriver
/// used to ensure that test ids are unique across drivers.
/// </summary>
string ID { get; set; }

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spaces are such a giant pain. I have been enjoying https://marketplace.visualstudio.com/items?itemName=MadsKristensen.TrailingWhitespaceVisualizer. When you save the file it helps by stripping all trailing whitespace.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Today I also discovered a VS Code extension for this too (https://marketplace.visualstudio.com/items?itemName=shardulm94.trailing-spaces) which is totally awesome. I enabled trim on save.

/// DirectTestRunner is the abstract base for runners
/// that deal directly with a framework driver.
/// </summary>
public class DirectTestRunner : AbstractTestRunner
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the parts in common could be abstracted to a base class as well, at some point, to help manage the complexity? Things like NUnitNetStandardDriverTests could then be consolidated and test all the drivers at once via a common interface at least.

<Compile Include="..\nunit.engine\Services\TestFilterBuilder.cs" Link="Services\TestFilterBuilder.cs" />
<Compile Include="..\nunit.engine\Services\TestFilterService.cs" Link="Services\TestFilterService.cs" />
<Compile Include="..\nunit.engine\Services\TestSelectionParser.cs" Link="Services\TestSelectionParser.cs" />
<Compile Include="..\nunit.engine\Services\Tokenizer.cs" Link="Services\Tokenizer.cs" />
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a glob could take care of almost all these items linked from the ..\nunit.engine folder. That's one less csproj to constantly be touching. :D (I can not wait until we do this with all the projects!)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, but this is just a fraction of the files in the Engine and API projects so I would have to add #if !NETSTANDARD1_3 to all of those. That may happen when/if the projects ever merge, but for now I wanted to keep this as unobtrusive as possible.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good strategy.

{
// Opening is enough to check
}
}

public void WriteResultFile(XmlNode resultNode, string outputPath)
{
using (var writer = new StreamWriter(outputPath, false, Encoding.UTF8))
using (var stream = new FileStream(outputPath, FileMode.Create, FileAccess.Write))
using (var writer = new StreamWriter(stream))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is better because it no longer writes a byte order mark.

<TargetFramework>netcoreapp1.1</TargetFramework>
<SignAssembly>True</SignAssembly>
<AssemblyOriginatorKeyFile>..\..\nunit.snk</AssemblyOriginatorKeyFile>
<DelaySign>False</DelaySign>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure this is the default and this line could be left out?

<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
<GenerateAssemblyProductAttribute>false</GenerateAssemblyProductAttribute>
<GenerateAssemblyTitleAttribute>false</GenerateAssemblyTitleAttribute>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same [These 9 lines can be replaced with <GenerateAssemblyInfo>false</GenerateAssemblyInfo>]

<Compile Include="..\nunit.engine.tests\Services\TestFilterBuilderTests.cs" Link="Services\TestFilterBuilderTests.cs" />
<Compile Include="..\nunit.engine.tests\Services\TestFilteringTests.cs" Link="Services\TestFilteringTests.cs" />
<Compile Include="..\nunit.engine.tests\Services\TestSelectionParserTests.cs" Link="Services\TestSelectionParserTests.cs" />
<Compile Include="..\nunit.engine.tests\Services\TokenizerTests.cs" Link="Services\TokenizerTests.cs" />
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same [Seems like a glob could take care of almost all these items linked from the ..\nunit.engine folder.]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, there are 53 CS files for the main test suite, we are only pulling in 14.

Copy link
Member Author

@rprouse rprouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jnm2 thanks for the review. I am making most of your project file change suggestions. Anything to clean up the files and make them easier to maintain is a win in my books 👍

<Compile Include="..\nunit.engine\Services\TestFilterBuilder.cs" Link="Services\TestFilterBuilder.cs" />
<Compile Include="..\nunit.engine\Services\TestFilterService.cs" Link="Services\TestFilterService.cs" />
<Compile Include="..\nunit.engine\Services\TestSelectionParser.cs" Link="Services\TestSelectionParser.cs" />
<Compile Include="..\nunit.engine\Services\Tokenizer.cs" Link="Services\Tokenizer.cs" />
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, but this is just a fraction of the files in the Engine and API projects so I would have to add #if !NETSTANDARD1_3 to all of those. That may happen when/if the projects ever merge, but for now I wanted to keep this as unobtrusive as possible.

<Compile Include="..\nunit.engine.tests\Services\TestFilterBuilderTests.cs" Link="Services\TestFilterBuilderTests.cs" />
<Compile Include="..\nunit.engine.tests\Services\TestFilteringTests.cs" Link="Services\TestFilteringTests.cs" />
<Compile Include="..\nunit.engine.tests\Services\TestSelectionParserTests.cs" Link="Services\TestSelectionParserTests.cs" />
<Compile Include="..\nunit.engine.tests\Services\TokenizerTests.cs" Link="Services\TokenizerTests.cs" />
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, there are 53 CS files for the main test suite, we are only pulling in 14.

<GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
<GenerateAssemblyProductAttribute>false</GenerateAssemblyProductAttribute>
<GenerateAssemblyTitleAttribute>false</GenerateAssemblyTitleAttribute>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That cleans things up, thanks. Do you know where all of the new options are documented? I've just been looking through examples of Microsoft projects.

@rprouse rprouse merged commit 3f71e38 into master Apr 28, 2017
@rprouse rprouse deleted the issue-10d branch April 28, 2017 20:05
@jnm2
Copy link
Collaborator

jnm2 commented Apr 28, 2017

@rprouse

That cleans things up, thanks. Do you know where all of the new options are documented? I've just been looking through examples of Microsoft projects.

Tell me about it. 😄
I still can't locate documentation for GenerateAssemblyInfo. Got it off dotnet/project-system.

I've used these recently:

https://docs.microsoft.com/en-us/dotnet/articles/core/tools/csproj
https://docs.microsoft.com/en-us/dotnet/articles/core/tools/project-json-to-csproj
http://www.natemcmaster.com/blog/2017/03/09/vs2015-to-vs2017-upgrade/
https://docs.microsoft.com/en-us/dotnet/articles/core/migration/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants