Split integration test build from test run #71166

dibarbet · 2023-12-08T01:50:54Z

This is part 1 of running integration tests on helix. This PR separates the build and testing stages in preparation for running on helix, it does not actually send the tests to helix to run.

This PR updates the integration test pipeline to use the same payload creation that we use for our helix unit tests. The integration test pipeline now does the following:

First build the assets once per configuration
Re-use the PrepareTests.csproj to create a minimized test assets payload containing the VSIX and integration test project outputs.
Upload that payload to the pipeline artifacts.
In a separate stage, we then download the test payload
Rehydrate the payload
Install the VSIXs and run the tests

In followup PRs, instead of running the integration tests on a normal ADO machine, we will instead partition them and upload them to helix, using the same payload support that I've added in this PR.

Example PR run:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=495861&view=results

Example CI run:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=496084&view=results

dibarbet · 2023-12-11T22:29:16Z

eng/build.ps1

-  $vsixDir = Get-PackageDir "RoslynTools.VSIXExpInstaller"
-  $vsixExe = Join-Path $vsixDir "tools\VsixExpInstaller.exe"
+
+  $vsixExe = Join-Path $ArtifactsDir "bin\Microsoft.VisualStudio.LanguageServices.New.IntegrationTests\$configuration\net472\VSIXExpInstaller\VSIXExpInstaller.exe"


Now that we're not building on the same machine, the nuget package directory is not available. Adding the vsix installer to the integration test project is an easy way to ensure that it is available in the test payload.

❗ This is an unsupported VSIX installer, and really needs to be removed from usage. The new integration tests already reference the new one, so we definitely should not be using that project to set this bad example. The solution here is to package up the solution packages folder separately from other build artifacts, similar to what I showed in the other example PR I sent you.

dibarbet · 2023-12-11T22:30:13Z

eng/pipelines/test-integration-helix.yml

+  - name: testRuns
+    type: object
+    default:
+      - oop64bit: true


the template accepts a list of input parameters so that we can re-use the build assets for multiple different configurations of test run.

dibarbet · 2023-12-11T22:31:45Z

eng/pipelines/test-integration-helix.yml

+        env:
+          HELIX_CORRELATION_PAYLOAD: '$(Build.SourcesDirectory)\.duplicate'
+
+      # This is a temporary step until the actual test run moves to helix (then we would need to rehydrate the tests there instead)


these will be removed in the next followup where the rehydration happens in the helix script.

dibarbet · 2023-12-11T22:32:30Z

eng/pipelines/test-integration-helix.yml

+          filePath: eng/build.ps1
+          arguments: -ci -prepareMachine -testVsi -configuration ${{ parameters.configuration }} -oop64bit:$${{ testParameters.oop64bit }} -oopCoreClr:$${{ testParameters.oopCoreClr }} -collectDumps -lspEditor:$${{ testParameters.lspEditor }}
+
+      # These are temporary publishing steps - once the tests run on helix, the artifacts will be attached to the helix payload.


These are basically unchanged from the normal template. Will be removed when on helix as we'll just copy the files into the helix upload root.

dibarbet · 2023-12-11T23:54:25Z

@sharwell @dotnet/roslyn-infrastructure PTAL

dibarbet · 2023-12-13T00:27:04Z

If I don't hear any further comments - planning on merging tomorrow.

sharwell · 2023-12-13T14:36:32Z

eng/build.ps1

-  $vsixDir = Get-PackageDir "RoslynTools.VSIXExpInstaller"
-  $vsixExe = Join-Path $vsixDir "tools\VsixExpInstaller.exe"
+
+  $vsixExe = Join-Path $ArtifactsDir "bin\Microsoft.VisualStudio.LanguageServices.New.IntegrationTests\$configuration\net472\VSIXExpInstaller\VSIXExpInstaller.exe"


❗ This is an unsupported VSIX installer, and really needs to be removed from usage. The new integration tests already reference the new one, so we definitely should not be using that project to set this bad example. The solution here is to package up the solution packages folder separately from other build artifacts, similar to what I showed in the other example PR I sent you.

eng/pipelines/test-integration-helix.yml

...est/New.IntegrationTests/Microsoft.VisualStudio.LanguageServices.New.IntegrationTests.csproj

dibarbet · 2023-12-14T18:56:51Z

@sharwell believe I've addressed your comments if you wouldn't mind taking another look

dotnet-issue-labeler bot added Area-Infrastructure untriaged Issues and PRs which have not yet been triaged by a lead labels Dec 8, 2023

dibarbet force-pushed the dev/dibarbet/integration_test_helix branch 15 times, most recently from 12a3423 to d13eb4f Compare December 8, 2023 22:15

dibarbet force-pushed the dev/dibarbet/integration_test_helix branch 3 times, most recently from 83a8c24 to 6589974 Compare December 11, 2023 19:12

build-analysis bot mentioned this pull request Dec 11, 2023

The active test run was aborted. Reason: Test host process crashed dotnet/dnceng#451

Open

3 tasks

dibarbet marked this pull request as ready for review December 11, 2023 22:17

dibarbet changed the title ~~[WIP] Split integration test build from test run~~ Split integration test build from test run Dec 11, 2023

dibarbet requested review from a team as code owners December 11, 2023 22:19

dibarbet commented Dec 11, 2023

View reviewed changes

arunchndr approved these changes Dec 12, 2023

View reviewed changes

sharwell requested changes Dec 13, 2023

View reviewed changes

sharwell reviewed Dec 13, 2023

View reviewed changes

eng/pipelines/test-integration-helix.yml Outdated Show resolved Hide resolved

sharwell reviewed Dec 13, 2023

View reviewed changes

eng/pipelines/test-integration-helix.yml Show resolved Hide resolved

sharwell reviewed Dec 13, 2023

View reviewed changes

eng/pipelines/test-integration-helix.yml Outdated Show resolved Hide resolved

sharwell reviewed Dec 13, 2023

View reviewed changes

...est/New.IntegrationTests/Microsoft.VisualStudio.LanguageServices.New.IntegrationTests.csproj Outdated Show resolved Hide resolved

Split integration test build from test run

365fc48

dibarbet force-pushed the dev/dibarbet/integration_test_helix branch from 6589974 to 365fc48 Compare December 14, 2023 00:52

dibarbet requested a review from sharwell December 14, 2023 05:34

sharwell approved these changes Dec 15, 2023

View reviewed changes

dibarbet merged commit 618b085 into main Dec 15, 2023

dibarbet deleted the dev/dibarbet/integration_test_helix branch December 15, 2023 21:38

ghost added this to the Next milestone Dec 15, 2023

allisonchou mentioned this pull request Dec 18, 2023

[Automated] PRs inserted in VS build main-34418.87 #71331

Closed

Cosifne modified the milestones: Next, 17.9 P3 Jan 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split integration test build from test run #71166

Split integration test build from test run #71166

dibarbet commented Dec 8, 2023 •

edited

Loading

dibarbet Dec 11, 2023

sharwell Dec 13, 2023

dibarbet Dec 11, 2023

dibarbet Dec 11, 2023

dibarbet Dec 11, 2023

dibarbet commented Dec 11, 2023

dibarbet commented Dec 13, 2023

sharwell Dec 13, 2023

dibarbet commented Dec 14, 2023

Split integration test build from test run #71166

Split integration test build from test run #71166

Conversation

dibarbet commented Dec 8, 2023 • edited Loading

dibarbet Dec 11, 2023

Choose a reason for hiding this comment

sharwell Dec 13, 2023

Choose a reason for hiding this comment

dibarbet Dec 11, 2023

Choose a reason for hiding this comment

dibarbet Dec 11, 2023

Choose a reason for hiding this comment

dibarbet Dec 11, 2023

Choose a reason for hiding this comment

dibarbet commented Dec 11, 2023

dibarbet commented Dec 13, 2023

sharwell Dec 13, 2023

Choose a reason for hiding this comment

dibarbet commented Dec 14, 2023

dibarbet commented Dec 8, 2023 •

edited

Loading