.Net Core 3.1 runtime breaks with csproj reference

[timcassell]

After fixing my issue with #1585, I'm getting an error only in the .Net Core 3.1 runtime. This only happens if I reference the csproj directly. If I build the netstandard 2.0 dll and reference it, it works fine in all runtimes. I also tried reverting the csproj to non-optimized and disabled the validator, but it still breaks.

I also noticed that it's saying the runtime is .Net 5.0, not sure why that is when I specified RuntimeMoniker.CoreRt31 as the runtime, not RuntimeMoniker.NetCoreApp50. I'm not sure if that has anything to do with my issue.

I'm on VS 16.7.7 and BenchmarkDotNet 0.12.1.

// Benchmark Process Environment Information:
// Runtime=.NET 5.0.29408.02 @BuiltBy: dlab14-DDVSOWINAGE075 @Branch: master @Commit: 4ce1c21ac0d4d1a3b7f7a548214966f69ac9f199, X64 AOT
// GC=Concurrent Workstation
// Job: Job-LAGRCI(RunStrategy=Throughput)

Error with csproj reference:

System.TypeLoadException: Could not load type 'Proto.Promises.Await.IAwaiter' from assembly 'ProtoPromise, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.
   at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowTypeLoadException(ExceptionStringID, String, String) + 0x11
   at AsynchronousBenchmarks.AwaitBenchmarks.<Proto>d__5.MoveNext() + 0x19
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&) + 0x3f
   at AsynchronousBenchmarks.AwaitBenchmarks.Proto() + 0x36
   at AsynchronousBenchmarks.AwaitBenchmarks.Proto_N() + 0x18
   at BenchmarkDotNet.Autogenerated.Runnable_2.WorkloadActionNoUnroll(Int64) + 0x2d
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData) + 0x1a3
   at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine, Int32, Int32, Int32) + 0x162
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters) + 0x14b
   at BenchmarkDotNet.Autogenerated.Runnable_2.Run(IHost, String) + 0x3cd
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[]) + 0x19b

[adamsitnik]

Hi @timcassell

This only happens if I reference the csproj directly

I've forked your code and investigated that. The problem that you describe was most probably caused by MsBuild configurations name mismatch. Between AsynchronousBenchmarks.csproj , ProtoPromise.csproj and the project with boilerplate code generated and build by BenchmarkDotNet.

AsynchronousBenchmarks.csproj project does not define any custom build configurations:

https://github.com/timcassell/CSharpAsynchronousBenchmarks/blob/b82ec7cfd04f5b44188a7b8ac7f6c0ef00e90170/AsynchronousBenchmarks/AsynchronousBenchmarks.csproj#L3-L6

while the referenced ProtoPromise.csproj does:

https://github.com/timcassell/ProtoPromise/blob/ab75d7981f989a82e40bca03413beb84f6332b22/Runtime/ProtoPromise.csproj#L6

BenchmarkDotNet by default compiles everything in Release mode, there was most probably a mismatch. (the code was compiling fine but various #if PROTO_PROMISE_PROGRESS_DISABLE blocks were determining what exactly was included)

I've sent a fix to your repo: timcassell/CSharpAsynchronousBenchmarks#2

The solution was to give up on custom msbuild configurations and use MsBuild property instead:

https://github.com/timcassell/CSharpAsynchronousBenchmarks/blob/38a88c11d7f66d0b0cf57254e345260e132803c3/AsynchronousBenchmarks/AsynchronousBenchmarks.csproj#L10

Now you can specify the TFMs via console line args:

dotnet run -c Release -f net472 --project .\AsynchronousBenchmarks\AsynchronousBenchmarks.csproj  --filter AsyncBenchmarks --runtimes net472 mono netcoreapp3.1 corert31

Sample output:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.1139 (1909/November2018Update/19H2)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
  [Host]     : .NET Framework 4.8 (4.8.4250.0), X64 RyuJIT
  Job-GGYYJO : .NET Framework 4.8 (4.8.4250.0), X64 RyuJIT
  Job-GQBLNP : .NET Core 3.1.7 (CoreCLR 4.700.20.36602, CoreFX 4.700.20.37001), X64 RyuJIT
  Job-YLPRJM : .NET 5.0.29408.02 @BuiltBy: dlab14-DDVSOWINAGE075 @Branch: master @Commit: 4ce1c21ac0d4d1a3b7f7a548214966f69ac9f199, X64 AOT
  Job-AUCLCO : Mono 5.10.0 (Visual Studio), X64

IterationCount=3  LaunchCount=1  WarmupCount=3

|     Method |        Job |       Runtime |             Toolchain |     N |         Mean |         Error |       StdDev | Ratio | RatioSD |     Gen 0 |    Gen 1 |    Gen 2 | Allocated |
|----------- |----------- |-------------- |---------------------- |------ |-------------:|--------------:|-------------:|------:|--------:|----------:|---------:|---------:|----------:|
|    Proto_N | Job-GGYYJO |    .NET 4.7.2 |                net472 |   100 |    201.67 us |    152.124 us |     8.338 us |  1.78 |    0.08 |   10.0098 |   4.8828 |   0.4883 |   64277 B |
|    Proto_A | Job-GGYYJO |    .NET 4.7.2 |                net472 |   100 |     60.35 us |     18.062 us |     0.990 us |  0.53 |    0.01 |         - |        - |        - |      32 B |
| SystemTask | Job-GGYYJO |    .NET 4.7.2 |                net472 |   100 |    113.18 us |      5.123 us |     0.281 us |  1.00 |    0.00 |   11.4746 |   1.0986 |        - |   73047 B |
|    UniTask | Job-GGYYJO |    .NET 4.7.2 |                net472 |   100 |     82.96 us |     10.817 us |     0.593 us |  0.73 |    0.00 |    0.4883 |   0.1221 |        - |    3794 B |
|    Proto_N | Job-GQBLNP | .NET Core 3.1 |         netcoreapp3.1 |   100 |    188.35 us |     63.592 us |     3.486 us |  1.66 |    0.03 |    8.0566 |   3.9063 |   0.4883 |   64088 B |
|    Proto_A | Job-GQBLNP | .NET Core 3.1 |         netcoreapp3.1 |   100 |     56.61 us |      7.908 us |     0.433 us |  0.50 |    0.00 |         - |        - |        - |      32 B |
| SystemTask | Job-GQBLNP | .NET Core 3.1 |         netcoreapp3.1 |   100 |     58.18 us |      7.095 us |     0.389 us |  0.51 |    0.00 |    7.1411 |   0.6104 |        - |   56032 B |
|    UniTask | Job-GQBLNP | .NET Core 3.1 |         netcoreapp3.1 |   100 |     77.13 us |     38.242 us |     2.096 us |  0.68 |    0.02 |         - |        - |        - |      32 B |
|    Proto_N | Job-YLPRJM |    CoreRt 3.1 | Core RT 1.0.0-alpha-* |   100 |    146.07 us |     21.600 us |     1.184 us |  1.29 |    0.01 |    7.3242 |   3.6621 |   0.2441 |   59288 B |
|    Proto_A | Job-YLPRJM |    CoreRt 3.1 | Core RT 1.0.0-alpha-* |   100 |     53.93 us |      6.609 us |     0.362 us |  0.48 |    0.00 |         - |        - |        - |      32 B |
| SystemTask | Job-YLPRJM |    CoreRt 3.1 | Core RT 1.0.0-alpha-* |   100 |     54.62 us |     14.882 us |     0.816 us |  0.48 |    0.01 |    6.5308 |   0.5493 |        - |   51232 B |
|    UniTask | Job-YLPRJM |    CoreRt 3.1 | Core RT 1.0.0-alpha-* |   100 |           NA |            NA |           NA |     ? |       ? |         - |        - |        - |         - |
|    Proto_N | Job-AUCLCO |          Mono |               Default |   100 |    359.00 us |    153.621 us |     8.421 us |  3.17 |    0.07 |   16.1133 |   2.4414 |   2.4414 |         - |
|    Proto_A | Job-AUCLCO |          Mono |               Default |   100 |     58.55 us |     11.475 us |     0.629 us |  0.52 |    0.00 |         - |        - |        - |         - |
| SystemTask | Job-AUCLCO |          Mono |               Default |   100 |     95.99 us |     29.194 us |     1.600 us |  0.85 |    0.01 |   20.8740 |        - |        - |         - |
|    UniTask | Job-AUCLCO |          Mono |               Default |   100 |     71.57 us |      1.895 us |     0.104 us |  0.63 |    0.00 |         - |        - |        - |         - |
|            |            |               |                       |       |              |               |              |       |         |           |          |          |           |
|    Proto_N | Job-GGYYJO |    .NET 4.7.2 |                net472 | 10000 | 28,490.25 us | 12,793.190 us |   701.238 us |  1.90 |    0.05 | 1093.7500 | 468.7500 | 187.5000 | 6419108 B |
|    Proto_A | Job-GGYYJO |    .NET 4.7.2 |                net472 | 10000 |  5,993.00 us |    173.365 us |     9.503 us |  0.40 |    0.00 |         - |        - |        - |         - |
| SystemTask | Job-GGYYJO |    .NET 4.7.2 |                net472 | 10000 | 15,021.72 us |    808.997 us |    44.344 us |  1.00 |    0.00 | 1156.2500 | 546.8750 |  46.8750 | 7301504 B |
|    UniTask | Job-GGYYJO |    .NET 4.7.2 |                net472 | 10000 |  8,614.68 us |  1,226.612 us |    67.235 us |  0.57 |    0.01 |   46.8750 |  15.6250 |        - |  376192 B |
|    Proto_N | Job-GQBLNP | .NET Core 3.1 |         netcoreapp3.1 | 10000 | 23,404.41 us |  7,403.596 us |   405.816 us |  1.56 |    0.03 |  937.5000 | 437.5000 | 156.2500 | 6400092 B |
|    Proto_A | Job-GQBLNP | .NET Core 3.1 |         netcoreapp3.1 | 10000 |  6,926.33 us | 18,677.302 us | 1,023.766 us |  0.46 |    0.07 |         - |        - |        - |         - |
| SystemTask | Job-GQBLNP | .NET Core 3.1 |         netcoreapp3.1 | 10000 |  7,455.47 us |  1,730.439 us |    94.851 us |  0.50 |    0.01 |  710.9375 | 281.2500 |        - | 5600032 B |
|    UniTask | Job-GQBLNP | .NET Core 3.1 |         netcoreapp3.1 | 10000 |  7,815.58 us |  1,465.230 us |    80.314 us |  0.52 |    0.01 |         - |        - |        - |         - |
|    Proto_N | Job-YLPRJM |    CoreRt 3.1 | Core RT 1.0.0-alpha-* | 10000 | 16,884.89 us |  1,209.419 us |    66.292 us |  1.12 |    0.01 |  843.7500 | 375.0000 | 125.0000 | 5920089 B |
|    Proto_A | Job-YLPRJM |    CoreRt 3.1 | Core RT 1.0.0-alpha-* | 10000 |  5,373.14 us |    295.425 us |    16.193 us |  0.36 |    0.00 |         - |        - |        - |         - |
| SystemTask | Job-YLPRJM |    CoreRt 3.1 | Core RT 1.0.0-alpha-* | 10000 |  6,782.08 us |  1,596.509 us |    87.510 us |  0.45 |    0.01 |  648.4375 | 304.6875 |        - | 5120032 B |
|    UniTask | Job-YLPRJM |    CoreRt 3.1 | Core RT 1.0.0-alpha-* | 10000 |           NA |            NA |           NA |     ? |       ? |         - |        - |        - |         - |
|    Proto_N | Job-AUCLCO |          Mono |               Default | 10000 | 44,362.03 us |  8,009.897 us |   439.049 us |  2.95 |    0.02 | 1583.3333 | 250.0000 | 250.0000 |         - |
|    Proto_A | Job-AUCLCO |          Mono |               Default | 10000 |  5,869.57 us |    699.452 us |    38.339 us |  0.39 |    0.00 |         - |        - |        - |         - |
| SystemTask | Job-AUCLCO |          Mono |               Default | 10000 | 14,862.36 us |  2,074.133 us |   113.690 us |  0.99 |    0.01 | 1812.5000 | 281.2500 | 281.2500 |         - |
|    UniTask | Job-AUCLCO |          Mono |               Default | 10000 |  7,506.12 us |  1,825.231 us |   100.047 us |  0.50 |    0.01 |         - |        - |        - |         - |

I also noticed that it's saying the runtime is .Net 5.0

This is another long story ;) BenchmarkDotNet needs to know the target framework moniker for which the benchmarks should be compiled. The problem with CoreRT is that no matter which TFM you provide, we always use latest version of it, which here got recognized as .NET 5. So the app was built for .NET Core 3.1 but run as CoreRT 5.0. This is related to how CoreRT apps are built: https://github.com/dotnet/corert/tree/master/samples/HelloWorld