Add support for user-supplied project file detection

[Genbox]

This update adds support for "Locators" in the IConfig interface. A Locator is an extensibility point where users can define path-handling logic that differs from the built-in logic.

For example, if the user sets AssemblyName in their csproj, they are forced to use the InProcess job, or else BenchmarkDotnet gives an error with:

Generate Exception: System.NotSupportedException: Unable to find in and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

The error is caused by the built-in logic trying to find a project file relative to the project root, which has the same name as the DLL. Unfortunately, lots of projects use AssemblyName to generate a short name, different from the project name (Like GNU utilities) or prepend the owners name (like GitHub/Nuget prefixes) to avoid assembly name collisions.

The issue has been repeatedly raised on the BenchmarkDotnet issue tracker and StackOverlow. For issues, see #1019, #498, #1178, #1785 and #1927.

The code

ILocator is a new public interface where users can define their logic. The interface is implemented like this:

private class CustomLocator : ILocator
{
    public LocatorType LocatorType => LocatorType.ProjectFile;

    public FileInfo Locate(DirectoryInfo rootDirectory, Type type)
    {
        return new FileInfo(Path.Combine(rootDirectory.FullName, @"Src\Benchmarks\MyBenchmarks.csproj"));
    }
}

The LocatorType is not strictly necessary, as the locator is only used in a single place right now. However, if there are other places where the user could benefit from the extensibility, it is necessary to have to distinguish between locators.

If that is not wanted, let me know, and I'll remove it and rename ILocator to IProjectLocator to specialize the code for its intended purpose.

Users can add their locator to a ManualConfig like so:

// To use both ProjectLocator.Default and custom locator:
IConfig config = ManualConfig.CreateMinimumViable()
                           .AddLocator(new CustomLocator());

// To use only the custom locator:
IConfig config = ManualConfig.CreateEmpty()
                           .AddLocator(new CustomLocator());

Notes

• ProjectLocator is a new class that contains the old logic to find the project file.
• DefaultConfig uses ProjectLocator.Default
• DebugConfig also uses ProjectLocator.Default
• ManualConfig defaults to no ILocator, but CreateMinimumViable includes ProjectLocator.Default
• I've added tests to ensure 1) duplicate locators are supported 2) that configs contains the correct locators

[timcassell]

Noting the breaking change to a public API.

[timcassell]

Also the DotNetCliGenerator.GetBuildArtifactsDirectoryPath may need to be updated to support the new locators.

[Genbox]

Most suggestions are now implemented.

[Genbox]

Update: ProjectLocator no longer exists. Instead, we simply give the user the ability to extend ManualConfig with a set of IFileLocators, which are executed before the default logic inside CsProjGenerator.GetProjectFilePath.

[Genbox]

For future reference and to show that it works.

I have a project that sets AssemblyName like this: <AssemblyName>Genbox.$(MSBuildProjectName)</AssemblyName>

With this code:

I get this error (as expected):

After changing it to use the new Locator API:

It works as expected:

[timcassell]

Can you please add an integration test?

[Genbox]

Can you please add an integration test?

The only solution I can think of is a separate project. Ideally, it would be a test inBenchmarkDotNet.IntegrationTests, but I cannot think of a way to do that. Do you have any ideas?

[timcassell]

Why can't you do it in BenchmarkDotNet.IntegrationTests? Just do like your example above:

fileInfo = new FileInfo(Path.Combine(dir, "../../../BenchmarkDotNet.IntegrationTests.csproj"));

[Genbox]

Why can't you do it in BenchmarkDotNet.IntegrationTests? Just do like your example above:

fileInfo = new FileInfo(Path.Combine(dir, "../../../BenchmarkDotNet.IntegrationTests.csproj"));

I would change the assembly name of BenchmarkDotNet.IntegrationTests.csproj, which will affect all the other tests. Adding a FileLocator that points to IntegrationTests.csproj could still work, even if FileLocators don't. The fallback will find the project instead, thereby hiding a bug.

So the alternative is to create a new integration test project like BenchmarkDotNet.IntegrationTests.ConfigPerAssembly and the others.

[timcassell]

Ah, that's a good point. Yes, you can create a new project and set it up like BenchmarkDotNet.IntegrationTests.ConfigPerAssembly.

[Genbox]

An integration test has been implemented.

[Genbox]

Please check that the csproj has the correct properties set. There were package ids and other stuff in the other projects that seemed irrelevant.

The tests also fails when running as debug. Not sure if this is intended or not. It is not something I've caused :)

[timcassell]

LGTM. We can update the GetBuildArtifactsDirectoryPath to support the new file locators separately.

Thanks @Genbox!

[Genbox]

LGTM. We can update the GetBuildArtifactsDirectoryPath to support the new file locators separately.

Thanks @Genbox!

Likewise! Thanks for your help and contributions. Very much appreciated.