Add GPU-enabled CI

[mattwthompson]

Fixes #

Changes made in this Pull Request:

PR Checklist

• Tests?
• Docs?
• CHANGELOG updated?
• Issue raised/referenced?

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.53%. Comparing base (350dcbd) to head (a6340d8).
Report is 3 commits behind head on main.

Additional details and impacted files

[mattwthompson]

The default disk allocation isn't enough for this conda environment. I checked with @ethanholz and the runner action doesn't expose an option for that yet. There might be a release (0.5.0?) in a week or so that includes this among other changes.

There might also be a way to slim dependencies down, I have not not looked into this yet.

[mattwthompson]

They made the release last Friday, upgrading to it is the next step here

[ethanholz]

As noted in our release notes, you will need to update the permissions on the AWS side to enable this functionality.

[mattwthompson]

I think I need somebody (@j-wags?) to add me into our AWS infrastructure to enable me to do that

The doc page is here: https://github.com/omsf-eco-infra/gha-runner/blob/v0.4.0/docs/aws.md

And the relevant change is: omsf-eco-infra/gha-runner@8be6d03

[j-wags]

I think I've updated this, could you let me know if it doesn't work?

[mattwthompson]

Unfortunately doesn't seem to be working - do we need to update something in the YAML?

[ethanholz]

Yes. As specified in the docs, you would need to choose a disk size larger than the default size of the AMI.

[mattwthompson]

Ah thanks, forgot about that

[mattwthompson]

@lilyminium something looks wrong with how Torch is set up here even though it's listed in the env, could you glance through briefly and see if something is obviously wrong?

[lilyminium]

I think that did it Matt 🎉 will the dgl channel have to keep getting updated?

[mattwthompson]

Wow ... I contexted-switched without coming back to see if the failures were genuine.

I do think the dgl channel will need to be updated - but I can live with that?

[lilyminium]

A few questions Matt:

• all the other CI was deleted -- was that just to help debugging or are we planning to move all CI onto GPU?
• I assume the intent is to run all tests on GPU, not just the one trial test?

[lilyminium]

Just took the liberty of reverting the additional test I added to check for CUDA availability.

[mattwthompson]

all the other CI was deleted

just forgot to put it back

I assume the intent is to run all tests on GPU, not just the one trial test?

This was intentional but only because that test was tagged as requiring a GPU. I guess we could run the entire test suite? (Are there other tests that are GPU-only in your memory?)

[mattwthompson]

Small quirk: since setup.cfg exists, this won't throw an error (until that changes, anyway) but it won't pick up any actual configs as they're in pyproject.toml. A leftover detail from #169

[mattwthompson]

Not sure why the other CI was failing earlier ... re-kicked in case it was just a bad channel. If all checks (except docs) are green this is good to go in my view

[lilyminium]

(Are there other tests that are GPU-only in your memory?)

Not in the test suite, but there was a gpu-only bug previously (#81) so it would be handy to run all tests to safeguard against that in the future!

[lilyminium]

LGTM -- thanks @mattwthompson for setting this up!!