Tests requiring internet access

[adamjstewart]

Description

Several of our tests require internet access. This is problematic for a number of reasons:

1. The GitHub Actions runners or the upload servers may have internet issues, leading to intermittent failures
2. Long download times slow down testing (the failing tests I discovered are all in our top 10 slowest tests)

We should apply the same magic we used in previous PRs to monkeypatch any downloads.

Steps to reproduce

1. Turn off WiFi or unplug the ethernet cable
2. Delete PyTorch's cached downloads: rm -r $(python -c 'import torch; print(torch.hub.get_dir())')
3. Run pytest

FAILED tests/models/test_farseg.py::TestFarSeg::test_valid_backbone[resnet18-True] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
FAILED tests/models/test_farseg.py::TestFarSeg::test_valid_backbone[resnet50-True] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
FAILED tests/trainers/test_detection.py::TestObjectDetectionTask::test_trainer[True-faster-rcnn-nasa_marine_debris-NASAMarineDebrisDataModule] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor ser...
FAILED tests/trainers/test_detection.py::TestObjectDetectionTask::test_trainer[True-fcos-nasa_marine_debris-NASAMarineDebrisDataModule] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname p...
FAILED tests/trainers/test_detection.py::TestObjectDetectionTask::test_trainer[True-retinanet-nasa_marine_debris-NASAMarineDebrisDataModule] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servn...
FAILED tests/trainers/test_detection.py::TestObjectDetectionTask::test_no_rgb[True] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
FAILED tests/trainers/test_detection.py::TestObjectDetectionTask::test_predict[True] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>

Version

0.5.0.dev0 (ce4e7ef)

[adamjstewart]

Bonus points if anyone can figure out how to jail our runners so that we can actually test this in CI and prevent it from slipping in again. We seem to fix bugs like this with every release.

Note that many of our slow tests (run only on release branches) are actually designed to test these kinds of dataset and weight downloads. We should be able to get 100% coverage without internet access, but can add additional slow tests to ensure that the weights still download correctly.

[adamjstewart]

Related to #194 (fixed in #265) and #495 (fixed in #497)

[adamjstewart]

@calebrob6 I think the detection tests could use the same treatment from #344, that would greatly speed things up.

[calebrob6]

Do you want to add that as an issue so we don't lose track of it, or should we just do it now? I guess we can just leave this open?

[adamjstewart]

It can be done at the same time, using a fake detection model and avoiding downloads. Both will speed up the test and make it more stable.

[adamjstewart]

FarSeg tests were failing on main because of this. Would love to get these fixed.

[adamjstewart]

Update: slowly whittling these down:

FAILED tests/models/test_farseg.py::TestFarSeg::test_valid_backbone[resnet18-True] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
FAILED tests/models/test_farseg.py::TestFarSeg::test_valid_backbone[resnet50-True] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
FAILED tests/trainers/test_detection.py::TestObjectDetectionTask::test_pretrained_backbone - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>

[adamjstewart]

Useful discussion (Linux-only):

• https://github.com/orgs/community/discussions/24975