CI: cache pip installs

[adamjstewart]

See https://github.com/actions/setup-python and https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#caching-packages

Edit by Caleb

See actions/setup-python#330 (comment) for a writeup from Adam about the speedup from this PR.

[adamjstewart]

Unclear how useful this is since the cache is invalidated any time these files change, and most of our PRs are dependabot PRs at the moment. It's probably most useful for the minimum dependencies which get updated very infrequently.

[calebrob6]

Do we have to do anything to skip the pip install after this?

[adamjstewart]

This is fun. It looks like cache only caches downloads, not installs. Compared to develop, avoiding these downloads shaves 25 secs off the "Install pip dependencies" step, but adds 35 secs to the "Set up python" step because it has to download and unpack the cache.

[calebrob6]

I'm not convinced the caching is doing the correct thing. E.g.

• if I look at the run of the ubuntu 3.8 tests on this PR I see that Python setup takes 29 seconds (it is downloading stuff from the cache), and pip install takes 2min 37 seconds
• if I look at the same run from the EuroSAT PR, Python setup takes 0 seconds and pip install takes 2min 36 seconds

It looks like the pip installs in this PR are using cached wheels somtimes (but, e.g., missing torch and downloading almost 1GB 😬 Downloading torch-1.13.1-cp38-cp38-manylinux1_x86_64.whl (887.4 MB))

[adamjstewart]

You beat me to it. Looks like there is an issue open for this: actions/setup-python#330

[adamjstewart]

So this seems slower than not caching. The only other benefit of caching is that it may help avoid download issues when a runner has internet access issues, but that seems infrequent enough that I would rather have fast installs. I vote we close this.

[calebrob6]

There should be a way to cache the install, else what is the point? (the only reason I looked so closely is because I think this is cool!)

[adamjstewart]

We could just use the cache action to cache everything: https://blog.allenai.org/python-caching-in-github-actions-e9452698e98d

[adamjstewart]

See actions/setup-python#330 (comment) for a matrix of install times. This is starting to look pretty good! We could still improve this by only caching the site-packages directory, but this varies by OS so I didn't try it here. Hopefully setup-python will add this feature someday, the current PR is a lot more lines than it was before.

[adamjstewart]

Other ideas for improving test times:

• Could also cache apt/brew/pacman installs, but these are fairly quick
• Could combine all style tests into a single test, resulting test would be slower than all other tests in parallel but would reduce the number of workers
• Could cache the run of pytest (this is a joke)
• Could try to speed up tests

This last one is most worthwhile imo. Our tests only take 2 min on Linux but up to 9 min on macOS/Windows for some reason. The detection trainer tests are particularly slow. There's definitely a lot of room for improvement here.

[calebrob6]

Definitely worthwhile to speed up tests -- do we have an open issue for this? If not can you make one?

[adamjstewart]

#1088 is for removing internet access and downloads which is the biggest low hanging fruit.

I'm about to open a series of PRs to hopefully speed up our tutorial tests. They only get run on release, which is why it was difficult to improve them (or bother caring about them).