-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add superpmi-replay pipeline file * Add superpmi-replay-job.yml that downloads collection * Moved superpmi-replay-job to right location * make sure to have spmi collection after jit build * annotate displayName with windows/OSX * Make osGroup=windows * Add helixType (if that matters) * Add some more variables * skip restore internal tools * Just use build-jit-job directly * Hardcode windows.x64.checked * Add -download_all -download_raw options to superpmi.py * Seperate 2 jobs * Fix Upload artifacts * Fix uploadAsArtifacts * Fix to uploadAsArtifacts * comment upload to azure storage temporarily * Set variables inside superpmi-replay-job * Entire pipeline * minor fix * Add some logging * Fix the path * fix file names and partition ids * add sample download on helix machine * fix helix-correlation-payload * Download specific build * Try to fix correlation payload directory * Do not run windows x86 for now * Log directory * walktree instead of dir * fix the bug * Add workitem directory as Payload * TEMP: Skip downloading collection during setup * fix core_root * Make download happen on helix * Add windows x86 * Use -target_os and pass 1 item to helix * Add some logging, have different work item ID * fix the superpmi.log upload * Fix the log location - Also include Linux_x64 * one more fix to log_directory * try to fix download results * Move the matrix to proj file * fix superpmi-replay.proj * Cleanup * Go back to download specific * WorkTimeout for 5 hours * Remove timeoutInMinutes * fix errors related to logging * return failed run result * fix the appending logic * Remove download-all and download-raw and instead add -no-progres * do not show download progress, try fixing download file * TO REVERT: Just try benchmark on windows x64/x86 * Use uploadAsArtifacts in build-jit-job * Create dummy.txt in workitem directory * create workitem_directory * Build clrjit * fix build-jit-job * add dependsOn * print params * TO REVERT: Just run jitbuild win-x64 * minor * move under jobParameters * Cleanup * review feedback
- Loading branch information
1 parent
9fc22a2
commit 0812322
Showing
13 changed files
with
641 additions
and
27 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| parameters: | ||
| unpackFolder: '' | ||
| cleanUnpackFolder: true | ||
| artifactFileName: '' | ||
| artifactName: '' | ||
| displayName: '' | ||
|
|
||
| steps: | ||
| # Download artifact | ||
| - task: DownloadBuildArtifacts@0 | ||
| displayName: 'Download specific ${{ parameters.displayName }}' | ||
| inputs: | ||
| buildType: specific | ||
| project: 'internal' # 'internal' or 'public' | ||
| pipeline: 'superpmi-replay' | ||
| buildVersionToDownload: specific | ||
| branchName: 'your/branch/having/artifacts' | ||
| buildId: '1274841' | ||
| downloadType: single | ||
| downloadPath: '$(Build.SourcesDirectory)/__download__' | ||
| artifactName: '${{ parameters.artifactName }}' | ||
|
|
||
| # Unzip artifact | ||
| - task: ExtractFiles@1 | ||
| displayName: 'Unzip specific ${{ parameters.displayName }}' | ||
| inputs: | ||
| archiveFilePatterns: $(Build.SourcesDirectory)/__download__/${{ parameters.artifactName }}/${{ parameters.artifactFileName }} | ||
| destinationFolder: ${{ parameters.unpackFolder }} | ||
| cleanDestinationFolder: ${{ parameters.cleanUnpackFolder }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| trigger: | ||
| batch: false | ||
| branches: | ||
| include: | ||
| - main | ||
| paths: | ||
| include: | ||
| - src/coreclr/jit/* | ||
| - src/coreclr/inc/jiteeversionguid.h | ||
|
|
||
| # This pipeline is supposed to be run only on merged changes | ||
| # and should not be triggerable from a PR. | ||
| pr: none | ||
|
|
||
| jobs: | ||
|
|
||
| - template: /eng/pipelines/common/platform-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/coreclr/templates/build-jit-job.yml | ||
| buildConfig: checked | ||
| platforms: | ||
| - windows_x64 | ||
| - windows_x86 | ||
| jobParameters: | ||
| uploadAs: 'pipelineArtifacts' | ||
|
|
||
| - template: /eng/pipelines/common/platform-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/coreclr/templates/superpmi-replay-job.yml | ||
| buildConfig: checked | ||
| platforms: | ||
| - windows_x64 | ||
| - windows_x86 | ||
| helixQueueGroup: ci | ||
| helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
125 changes: 125 additions & 0 deletions
125
eng/pipelines/coreclr/templates/run-superpmi-replay-job.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| parameters: | ||
| steps: [] # optional -- any additional steps that need to happen before pulling down the jitutils repo and sending the jitutils to helix (ie building your repo) | ||
| variables: [] # optional -- list of additional variables to send to the template | ||
| jobName: '' # required -- job name | ||
| displayName: '' # optional -- display name for the job. Will use jobName if not passed | ||
| pool: '' # required -- name of the Build pool | ||
| container: '' # required -- name of the container | ||
| buildConfig: '' # required -- build configuration | ||
| archType: '' # required -- targeting CPU architecture | ||
| osGroup: '' # required -- operating system for the job | ||
| osSubgroup: '' # optional -- operating system subgroup | ||
| extraSetupParameters: '' # optional -- extra arguments to pass to the setup script | ||
| frameworks: ['netcoreapp3.0'] # optional -- list of frameworks to run against | ||
| continueOnError: 'false' # optional -- determines whether to continue the build if the step errors | ||
| dependsOn: '' # optional -- dependencies of the job | ||
| timeoutInMinutes: 320 # optional -- timeout for the job | ||
| enableTelemetry: false # optional -- enable for telemetry | ||
| liveLibrariesBuildConfig: '' # optional -- live-live libraries configuration to use for the run | ||
| runtimeType: 'coreclr' # optional -- Sets the runtime as coreclr or mono | ||
| codeGenType: 'JIT' # optional -- Decides on the codegen technology if running on mono | ||
| runKind: '' # required -- test category | ||
| collectionType: '' | ||
| collectionName: '' | ||
| dependOnEvaluatePaths: false | ||
|
|
||
| jobs: | ||
| - template: xplat-pipeline-job.yml | ||
| parameters: | ||
| dependsOn: ${{ parameters.dependsOn }} | ||
| buildConfig: ${{ parameters.buildConfig }} | ||
| archType: ${{ parameters.archType }} | ||
| osGroup: ${{ parameters.osGroup }} | ||
| osSubgroup: ${{ parameters.osSubgroup }} | ||
| liveLibrariesBuildConfig: ${{ parameters.liveLibrariesBuildConfig }} | ||
| enableTelemetry: ${{ parameters.enableTelemetry }} | ||
| enablePublishBuildArtifacts: true | ||
| continueOnError: ${{ parameters.continueOnError }} | ||
| collectionType: $ {{ parameters.collectionType }} | ||
| collectionName: ${{ parameters.collectionName }} | ||
| dependOnEvaluatePaths: ${{ parameters.dependOnEvaluatePaths }} | ||
| timeoutInMinutes: ${{ parameters.timeoutInMinutes }} | ||
|
|
||
| ${{ if ne(parameters.displayName, '') }}: | ||
| displayName: '${{ parameters.displayName }}' | ||
| ${{ if eq(parameters.displayName, '') }}: | ||
| displayName: '${{ parameters.jobName }}' | ||
|
|
||
| variables: | ||
| - ${{ each variable in parameters.variables }}: | ||
| - ${{ if ne(variable.name, '') }}: | ||
| - name: ${{ variable.name }} | ||
| value: ${{ variable.value }} | ||
| - ${{ if ne(variable.group, '') }}: | ||
| - group: ${{ variable.group }} | ||
|
|
||
| - name: PythonScript | ||
| value: 'py -3' | ||
| - name: PipScript | ||
| value: 'py -3 -m pip' | ||
| - name: SpmiCollectionLocation | ||
| value: '$(Build.SourcesDirectory)\artifacts\spmi\' | ||
| - name: SpmiLogsLocation | ||
| value: '$(Build.SourcesDirectory)\artifacts\spmi_logs\' | ||
| - name: HelixResultLocation | ||
| value: '$(Build.SourcesDirectory)\artifacts\helixresults\' | ||
|
|
||
| workspace: | ||
| clean: all | ||
| pool: | ||
| ${{ parameters.pool }} | ||
| container: ${{ parameters.container }} | ||
| strategy: | ||
| matrix: | ||
| ${{ each framework in parameters.frameworks }}: | ||
| ${{ framework }}: | ||
| _Framework: ${{ framework }} | ||
| steps: | ||
| - ${{ parameters.steps }} | ||
|
|
||
| - script: | | ||
| mkdir -p $(SpmiCollectionLocation) | ||
| displayName: Create directory for SPMI collection | ||
| - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/superpmi_replay_setup.py -source_directory $(Build.SourcesDirectory) -product_directory $(buildProductRootFolderPath) -arch $(archType) | ||
| displayName: ${{ format('SuperPMI replay setup ({0} {1})', parameters.osGroup, parameters.archType) }} | ||
|
|
||
| # Run superpmi replay in helix | ||
| - template: /eng/pipelines/coreclr/templates/superpmi-send-to-helix.yml | ||
| parameters: | ||
| HelixSource: '$(HelixSourcePrefix)/$(Build.Repository.Name)/$(Build.SourceBranch)' # sources must start with pr/, official/, prodcon/, or agent/ | ||
| HelixAccessToken: $(HelixApiAccessToken) | ||
| HelixTargetQueues: $(Queue) | ||
| HelixPreCommands: $(HelixPreCommand) | ||
| Creator: $(Creator) | ||
| WorkItemTimeout: 4:00 # 4 hours | ||
| WorkItemDirectory: '$(WorkItemDirectory)' | ||
| CorrelationPayloadDirectory: '$(CorrelationPayloadDirectory)' | ||
| ProjectFile: 'superpmi-replay.proj' | ||
| BuildConfig: ${{ parameters.buildConfig }} | ||
| osGroup: ${{ parameters.osGroup }} | ||
| archType: ${{ parameters.archType }} | ||
| continueOnError: true # Run the future step i.e. upload superpmi logs | ||
|
|
||
| # Always upload the available logs for diagnostics | ||
| - task: CopyFiles@2 | ||
| displayName: Copying superpmi.log of all partitions | ||
| inputs: | ||
| sourceFolder: '$(HelixResultLocation)' | ||
| contents: '**/superpmi_*.log' | ||
| targetFolder: '$(SpmiLogsLocation)' | ||
| condition: always() | ||
|
|
||
| - task: PublishPipelineArtifact@1 | ||
| displayName: Publish Superpmi logs | ||
| inputs: | ||
| targetPath: $(SpmiLogsLocation) | ||
| artifactName: 'SuperPMI_Logs_$(archType)_$(buildConfig)' | ||
| condition: always() | ||
|
|
||
| - task: PublishPipelineArtifact@1 | ||
| displayName: Publish SuperPMI build logs | ||
| inputs: | ||
| targetPath: $(Build.SourcesDirectory)/artifacts/log | ||
| artifactName: 'SuperPMI_BuildLogs__$(archType)_$(buildConfig)' | ||
| condition: always() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| parameters: | ||
| buildConfig: '' # required -- build configuration | ||
| archType: '' # required -- targeting CPU architecture | ||
| osGroup: '' # required -- operating system for the job | ||
| osSubgroup: '' # optional -- operating system subgroup | ||
| pool: '' | ||
| stagedBuild: false | ||
| timeoutInMinutes: 320 # build timeout | ||
| framework: net5.0 # Specify the appropriate framework when running release branches (ie netcoreapp3.0 for release/3.0) | ||
| variables: {} | ||
| dependOnEvaluatePaths: false | ||
| runJobTemplate: '/eng/pipelines/coreclr/templates/run-superpmi-replay-job.yml' | ||
|
|
||
| jobs: | ||
| - template: ${{ parameters.runJobTemplate }} | ||
| parameters: | ||
| jobName: ${{ format('superpmibuild_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }} | ||
| displayName: ${{ format('SuperPMI replay {0} {1}', parameters.osGroup, parameters.archType) }} | ||
| pool: ${{ parameters.pool }} | ||
| buildConfig: ${{ parameters.buildConfig }} | ||
| archType: ${{ parameters.archType }} | ||
| osGroup: ${{ parameters.osGroup }} | ||
| osSubgroup: ${{ parameters.osSubgroup }} | ||
| dependOnEvaluatePaths: ${{ parameters.dependOnEvaluatePaths }} | ||
| timeoutInMinutes: ${{ parameters.timeoutInMinutes }} | ||
| additionalSetupParameters: ${{ parameters.additionalSetupParameters }} | ||
| dependsOn: | ||
| - ${{ format('coreclr_jit_build_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }} | ||
|
|
||
| variables: ${{ parameters.variables }} | ||
|
|
||
| frameworks: | ||
| - ${{ parameters.framework }} | ||
|
|
||
| steps: | ||
|
|
||
| # Download jit builds | ||
| - template: /eng/pipelines/common/download-artifact-step.yml | ||
| parameters: | ||
| unpackFolder: $(buildProductRootFolderPath) | ||
| artifactFileName: '$(buildProductArtifactName)$(archiveExtension)' | ||
| artifactName: '$(buildProductArtifactName)' | ||
| displayName: 'JIT product build' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.