best-practices devops: add nightly_gpu_tests pipeline #823
Conversation
bethz
requested review from
gramhagen,
miguelgfierro and
yueguoguo
as code owners
tests/ci/nightly_gpu.yml
Outdated
| test: 'tests/ci/run_pytest.py' | ||
| maxnodes : 4 | ||
| reponame : 'Recommenders' | ||
| branch : 'azure-pipelines-bz' |
There was a problem hiding this comment.
this should be in staging (or master, when it is computed in master), right?
There was a problem hiding this comment.
Currently, it's only a string that is tagged in an AML experiment for correlation. Eventually, adding code to use the actual pipeline would be useful.
I changed it so something more generic for now.
tests/ci/nightly_gpu.yml
Outdated
|
|
||
| trigger: none | ||
|
|
||
| pr: |
There was a problem hiding this comment.
Nightly tests should be triggered every day, not when some does a PR (as opposite to unit tests)
There was a problem hiding this comment.
pr trigger removed from yml.
The nightly trigger will be set in the devops pipeline trigger setting after the PR is in staging.
| @@ -0,0 +1,108 @@ | |||
| # Copyright (c) Microsoft Corporation. All rights reserved. | |||
There was a problem hiding this comment.
do you think you will have to change the name to not get a conflict?
There was a problem hiding this comment.
the pipeline name is the conflict not the yml.
When the PR is included in staging, then the pipeline name is changed to bp-nightly_gpu and the yml file remains the same.
| - template: env-setup.yml # template reference | ||
|
|
||
| - script: | ||
| python scripts/generate_conda_file.py --gpu --name reco |
There was a problem hiding this comment.
there is no need to reinstall the libraries again. The smoke and integration tests in the pipeline that we have in devops use the same environment
There was a problem hiding this comment.
Smoke and integration are currently set up as separate jobs for devops which means if there is a bug in one of the integration tests, this set up makes it easy to rerun just the integration tests without needing to run smoke. Smoke tests take about 22m and integration takes just under 1.5 hours. The lib install is about 1.5 min.
I'm currently testing a version where I remove those from the integration tests. I'll update the pr once it runs.
There was a problem hiding this comment.
The integration test and smoke test are in separate jobs in the same pipeline. The smoke test must run successfully before the integration job will run but the integration job does not share setup with the smoke test as they are in separate jobs. I removed the installs in the integration test and ran the integration test after the smoke test. The integration test failed when the env-setup.yml was not run. I tested this once. The az login was also required.
|
merging now, test failure is not related |
best-practices devops: add nightly_gpu_tests pipeline
Description
This pipeline adds nightly_gpu_tests to the best-practices devops instance.
Related Issues
This is new work.
Checklist: