SonarQube.Integration.ImportBefore.targets removal by TargetsUninstaller causes issues with Concurrent Analysis

[japj]

Description

In a situation with concurrent analysis on the same build machine, the TargetUninstaller will always remove the SonarQube.Integration.ImportBefore.targets from its installation folder, causing issues with "file in use".

Repro steps

1. Setup a machine with concurrent analysis, where one build finishes while the other is still running
2. Run lots of concurrent builds
3. Depending on timing, this can cause:

The process cannot access the file 'C:\Users\<username>\AppData\Local\Microsoft\MSBuild\Current\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets' because it is being used by another process.

Expected behavior

In a concurrent analysis environment, TargetUninstaller should NOT uninstall the SonarQube.Integration.ImportBefore.targets

Actual behavior

 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\4.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\10.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\10.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\11.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\11.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\12.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  SonarQube.Integration.ImportBefore.targets does not exist at C:\Users\<username>\AppData\Local\Microsoft\MSBuild\12.0\Microsoft.Common.targets\ImportBefore
 01:36:12.334  01:36:12.334  SonarQube.Integration.ImportBefore.targets does not exist at C:\Users\<username>\AppData\Local\Microsoft\MSBuild\14.0\Microsoft.Common.targets\ImportBefore
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\14.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\15.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\15.0\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\Current\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets
 01:36:12.334  01:36:12.334  Uninstalling target: C:\Users\<username>\AppData\Local\Microsoft\MSBuild\Current\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets

Known workarounds

none

Related information

• SonarScanner for MSBuild version: 4.6.2
• SonarQube server version: 6.7.5
• Languages in your solution : C#, C++

[mickael-caro-sonarsource]

Hi,

This scenario is not supported by default, however, have you followed the workaround proposed here ? (In the Advanced topics part) https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/

Thanks.

[japj]

Hi Mickael, I will have another look at following the instructions and report if we still run into issues (although I thought we already did that).

However, I noted there is a little bit of outdated/incorrect information in the docs you linked:

• For v16, default path is: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Microsoft.Common.targets*ImportBefore* (ImportBefore is missing)
• For dotnet, the path is <DOTNET_SDK_INSTALL_DIR>*Current*\Microsoft.Common.targets\ImportBefore (15.0 should be replaced with Current for recent dotnet sdk)

[japj]

@mickael-caro-sonarsource , unfortunately even with the instruction of copying files to the global VS ImportBefore folder, SonarScanner.MSBuild.exe begin will still copy files to the local user ImportBefore folders. ("11:21:12.182 Updating build integration targets...")

This causes MSBuild to actually look for and import also the per-user ImportBefore file.
It is this per-user import that is causing the problem in combination with running SonarScanner.MSBuild.exe end, since that is actually Uninstalling these per-user files at a moment in time when another (concurrent) MSbuild is happening.

We see this issue occurring at least once a week across a range of builds on our servers.

[mickael-caro-sonarsource]

Thanks @japj , we'll have a look at that.

[japj]

@mickael-caro-sonarsource hope you are all ok.
Do you need any additional help with analysis or determine a good solution path for this?

[cjquintero]

This scenario is not supported by default, however, have you followed the workaround proposed here ? (In the Advanced topics part) https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/

The workaround is not really good because:

1. Concurrency should be provided "out of the box". It is quite usual to have a Jenkins server with two build executors and the following timing sequence ruins the second job:

• Job1: SonarScanner.MSBuild.exe begin
• Job1: MSBuild.exe
• Job2: SonarScanner.MSBuild.exe begin
• Job1: SonarScanner.MSBuild.exe end <--- This DELETES SonarQube.Integration.ImportBefore.targets
• Job2: MSBuild.exe <-- Builds OK, but without SonarQube.Integration.ImportBefore.targets
• Job2: SonarScanner.MSBuild.exe end <-- Causes "The SonarQube MSBuild integration failed: SonarQube was unable to collect the required information about your projects."

(notice that I don't get the error reported by japj "The process cannot access the file 'C:\Users<username>\AppData\Local\Microsoft\MSBuild\Current\Microsoft.Common.targets\ImportBefore\SonarQube.Integration.ImportBefore.targets' because it is being used by another process" but the error I get is fully reproducible with two .bat files with PAUSE statements after each command.

2. The workaround causes duplicated SonarQube.Integration.ImportBefore.targets injection, which is not very "clean".

3. The workaround ruins the scenario where you want two builds (from the docs: "... WarningsAsErrors will be turned off. If your build process cannot tolerate these changes we recommend creating a second build job for SonarQube analysis."):

• The first one to use TreatWarningsAsErrors = True, with no SonarQube involved, but the SonarQube.Integration.ImportBefore.targets always present ruins it because SonarQube sets TreatWarningsAsErrors = False
• The second one with SonarQube involved.

[cjquintero]

Do you need any additional help with analysis or determine a good solution path for this?

Try the following approach:

• SonarScanner.MSBuild.exe begin ...
• DEL "%USERPROFILE%\AppData\Local\Microsoft\MSBuild\14.0\Microsoft.Common.targets\ImportBefore
  \SonarQube.Integration.ImportBefore.targets" (at this point you can control the deleting error if any, with delayed retries, etc.)
• msbuild.exe /t:Rebuild ... /p:CustomBeforeMicrosoftCSharpTargets="%SONAR_FOLDER%\Targets\SonarQube.Integration.ImportBefore.targets"
• DEL "%USERPROFILE%\AppData\Local\Microsoft\MSBuild\14.0\Microsoft.Common.targets\ImportBefore
  \SonarQube.Integration.ImportBefore.targets" (at this point you can control the deleting error if any, with delayed retries, etc.)
• SonarScanner.MSBuild.exe end ...

This approach:

• Minimizes the time while the file SonarQube.Integration.ImportBefore.targets exists in the folder "%USERPROFILE%\AppData\Local\Microsoft\MSBuild\14.0\Microsoft.Common.targets\ImportBefore"
• Allows you to control errors while deleting that file after SonarScanner.MSBuild.exe begin and before the msbuild and (just in case) after the msbuild
• Injects the target "SonarQube.Integration.ImportBefore.targets" directly in the msbuild step with the MSBuild property named CustomBeforeMicrosoftCSharpTargets

[japj]

@cjquintero unfortunately, your workaround does not solve the problem we are having with the The process cannot access the file.

The problem is that the SonarScanner.MSBuild.exe will still copy and remove files in the global importbefore folders where MSBuild will pick them up (regardless if whether CustomBeforeMicrosoftCSharpTargets is set).

The ImportBefore folders are actually handled by Microsoft.Common.props and the CustomBeforeMicrosoftCSharpTargets is handled by Microsoft.CSharp.targets

[cjquintero]

@japj Yes, but SonarScanner.MSBuild.exe copies and remove the files in different phases: they are copied in the "begin" phase and are removed in the "end" phase. In between, you have two chances of removing them in your build (steps 2 and 4 of my list) handling the error "The process cannot access the file", for example, with retries. If you manage to delete the files before the step 5 of my list, when the "end" phase of the SonarScanner.MSBuild.exe is going to delete them they no longer exist. The difference is that you can control the exception deleting the file and retry a number of times until it succeeds.

That said, I have never seen the error "The process cannot access the file" (maybe because I don't have "lots" of concurrent builds) but I have suffered a concurrency problem ("SonarQube was unable to collect the required information about your projects") with two simultaneous Jenkins builds (that I explained above) and I managed to solve it with my workaround.

And finally, I think you can disable the use of "%USERPROFILE%\AppData\Local\Microsoft\MSBuild\14.0\Microsoft.Common.targets\ImportBefore", so although SonarScanner.MSBuild.exe copies/deletes files there, MSBuild doesn't use them, minimizing the chances of "access denied". This could be done as explained here: https://docs.microsoft.com/en-us/visualstudio/msbuild/customize-your-build ("MSBuildExtensionsPath and MSBuildUserExtensionsPath" section).

[japj]

@mickael-caro-sonarsource great news! can't wait to rollout, do you have any e.t.a. on the 5.1.0 release?

[mlaily]

Hello,
Does this issue being fixed mean concurrent analyses will now work out of the box without having to copy targets files to the global MSbuild targets?

(Or is the workaround still required? https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/ does not seem to have been updated but I'm not sure it's on purpose)

---

EDIT: Ok, so I went ahead and tested this, and it looks like the begin command copies SonarQube.Integration.ImportBefore.targets to all the MSBuild installation locations it can find, and end doesn't remove them anymore.

Correct me if I'm wrong, but it looks like concurrent builds are not a problem anymore (at the price of targets files always remaining on the build machine), and https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/ should be updated accordingly.

[mickael-caro-sonarsource]

You're right @mlaily thanks for the heads up.