Add GPU CI/CD #253

sarahyurick · 2024-09-18T21:12:28Z

This PR enables gpuCI for our PyTests marked with @pytest.mark.gpu.

To trigger it, a user has to add the gpuci label to their PR for the GPU tests to run. Only users with write access are able to add labels to PRs.

If more commits are added to the PR, the gpuci label has to be removed and re-added to run the GPU tests again. This is for security reasons; for example, a random user could open a PR, a user with write access adds the gpuci label, the GPU tests run safely, and then the user pushes malicious code. As long as no one with write access re-adds the gpuci label, it won't run at all.

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick

Need to fix trailing whitespaces in gpuci.yml and should make the spacing in auto-label.yml consistent with gpuci.yml.

Also, maybe I should rename it to auto-label-gpuci.yml.

.github/workflows/auto-label.yml

.github/workflows/gpuci.yml

sarahyurick · 2024-09-18T21:44:45Z

.github/workflows/gpuci.yml

+
+    - name: Run PyTests with GPU mark
+      run: |
+        python -m pytest -m gpu


We run our GPU CI tests only on PyTests with the pytest.mark.gpu decorator. We already have quite a few of them in our repository:

exact_dedup_data

TestSemDuplicates

TestMetadataSep

TestMinhashes

TestLSH

TestFuzzyDuplicates

test_extract_partitioning_index

.github/workflows/auto-label.yml

Signed-off-by: Sarah Yurick <[email protected]>

.github/workflows/gpuci.yml

README.md

.github/workflows/gpuci.yml

README.md

Signed-off-by: Sarah Yurick <[email protected]>

praateekmahajan

This is great and helps us catch issues much earlier! Thank you for pushing through this!

Dockerfile

.github/workflows/gpuci.yml

ayushdg

Minor suggestions

Dockerfile

.github/workflows/gpuci.yml

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick · 2024-10-07T22:19:16Z

@ko3n1g is disconnecting our runner for a bit. Should be back up tomorrow, then I can rerun the checks and merge.

Signed-off-by: Sarah Yurick <[email protected]>

ko3n1g

This is important since it impacts other projects that use this infra

Dockerfile

Signed-off-by: Sarah Yurick <[email protected]>

Addressed review

* add yaml files to gh workflows Signed-off-by: Sarah Yurick <[email protected]> * edit spacing Signed-off-by: Sarah Yurick <[email protected]> * no cache dir Signed-off-by: Sarah Yurick <[email protected]> * cmake Signed-off-by: Sarah Yurick <[email protected]> * fasttext wheel Signed-off-by: Sarah Yurick <[email protected]> * python3 dev Signed-off-by: Sarah Yurick <[email protected]> * get update Signed-off-by: Sarah Yurick <[email protected]> * c installs Signed-off-by: Sarah Yurick <[email protected]> * setuptools pip upgrade Signed-off-by: Sarah Yurick <[email protected]> * use stable rapids Signed-off-by: Sarah Yurick <[email protected]> * remove wheel see what happens Signed-off-by: Sarah Yurick <[email protected]> * edit readme and remove autolabel for now Signed-off-by: Sarah Yurick <[email protected]> * add container logic Signed-off-by: Sarah Yurick <[email protected]> * add dockerfile and oliver's other suggestions Signed-off-by: Sarah Yurick <[email protected]> * fix run format Signed-off-by: Sarah Yurick <[email protected]> * forked repo url Signed-off-by: Sarah Yurick <[email protected]> * docker run with all gpus Signed-off-by: Sarah Yurick <[email protected]> * remove running container Signed-off-by: Sarah Yurick <[email protected]> * Update .github/workflows/gpuci.yml Co-authored-by: oliver könig <[email protected]> Signed-off-by: Sarah Yurick <[email protected]> * re add test Signed-off-by: Sarah Yurick <[email protected]> * debug attempt Signed-off-by: Sarah Yurick <[email protected]> * remove it Signed-off-by: Sarah Yurick <[email protected]> * add library path Signed-off-by: Sarah Yurick <[email protected]> * remove nvcc check Signed-off-by: Sarah Yurick <[email protected]> * more debugging Signed-off-by: Sarah Yurick <[email protected]> * specify curator dir Signed-off-by: Sarah Yurick <[email protected]> * more debugging Signed-off-by: Sarah Yurick <[email protected]> * try pytorch container Signed-off-by: Sarah Yurick <[email protected]> * use rapids container Signed-off-by: Sarah Yurick <[email protected]> * fix RUN instructions Signed-off-by: Sarah Yurick <[email protected]> * add comments and review suggestions Signed-off-by: Sarah Yurick <[email protected]> * update runners Signed-off-by: Sarah Yurick <[email protected]> * move args Signed-off-by: Sarah Yurick <[email protected]> --------- Signed-off-by: Sarah Yurick <[email protected]> Signed-off-by: Sarah Yurick <[email protected]> Co-authored-by: oliver könig <[email protected]> Signed-off-by: Vinay Raman <[email protected]>

add yaml files to gh workflows

d560fe1

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick commented Sep 18, 2024

View reviewed changes

.github/workflows/auto-label.yml Outdated Show resolved Hide resolved

edit spacing

a5f5da8

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added the gpuci Run GPU CI/CD on PR label Sep 19, 2024

no cache dir

fe8ba83

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Sep 19, 2024

cmake

a1bfd97

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Sep 19, 2024

fasttext wheel

e1a47e4

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Sep 19, 2024

python3 dev

da6d552

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Sep 19, 2024

get update

24ba153

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Sep 19, 2024

c installs

167cefc

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Sep 19, 2024

ayushdg reviewed Sep 19, 2024

View reviewed changes

.github/workflows/gpuci.yml Outdated Show resolved Hide resolved

.github/workflows/gpuci.yml Outdated Show resolved Hide resolved

.github/workflows/gpuci.yml Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

praateekmahajan reviewed Sep 19, 2024

View reviewed changes

.github/workflows/gpuci.yml Outdated Show resolved Hide resolved

praateekmahajan reviewed Sep 19, 2024

View reviewed changes

README.md Show resolved Hide resolved

setuptools pip upgrade

a729c2c

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Sep 19, 2024

use stable rapids

db2e12d

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Oct 7, 2024

fix RUN instructions

06a2def

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Oct 7, 2024

praateekmahajan approved these changes Oct 7, 2024

View reviewed changes

praateekmahajan reviewed Oct 7, 2024

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

praateekmahajan reviewed Oct 7, 2024

View reviewed changes

.github/workflows/gpuci.yml Outdated Show resolved Hide resolved

ayushdg previously requested changes Oct 7, 2024

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

Dockerfile Outdated Show resolved Hide resolved

Dockerfile Outdated Show resolved Hide resolved

ayushdg reviewed Oct 7, 2024

View reviewed changes

.github/workflows/gpuci.yml Outdated Show resolved Hide resolved

add comments and review suggestions

8b5acff

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Oct 7, 2024

update runners

8f50714

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Oct 8, 2024

sarahyurick requested review from VibhuJawa, praateekmahajan and ayushdg October 8, 2024 20:03

ko3n1g requested changes Oct 9, 2024

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

move args

a9dd8e5

Signed-off-by: Sarah Yurick <[email protected]>

sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Oct 9, 2024

ko3n1g approved these changes Oct 9, 2024

View reviewed changes

sarahyurick merged commit 0bbdc06 into NVIDIA:main Oct 9, 2024
5 checks passed

sarahyurick deleted the gpuci branch October 25, 2024 20:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GPU CI/CD #253

Add GPU CI/CD #253

sarahyurick commented Sep 18, 2024 •

edited

Loading

sarahyurick left a comment

sarahyurick Sep 18, 2024

praateekmahajan left a comment

ayushdg left a comment

sarahyurick commented Oct 7, 2024

ko3n1g left a comment

Add GPU CI/CD #253

Add GPU CI/CD #253

Conversation

sarahyurick commented Sep 18, 2024 • edited Loading

sarahyurick left a comment

Choose a reason for hiding this comment

sarahyurick Sep 18, 2024

Choose a reason for hiding this comment

praateekmahajan left a comment

Choose a reason for hiding this comment

ayushdg left a comment

Choose a reason for hiding this comment

sarahyurick commented Oct 7, 2024

ko3n1g left a comment

Choose a reason for hiding this comment

sarahyurick commented Sep 18, 2024 •

edited

Loading