Test fewer models in trainers to avoid exceeding RAM

[calebrob6]

We don't need to test the BYOL trainer with every set of pretrained weights (we don't actually need to involve pretrained weights at all).

[adamjstewart]

From #1376:

We might be able to only load 1 ResNet, but we'll still have to load all of them on the release branch, so we can't just avoid the problem.

We want to make sure our weights actually load in a model correctly. All of them. Every time someone adds a new one.

[calebrob6]

We certainly don't need to check the cross product of pretrained weights with every trainer as there will be a lot of duplicate work (e.g. we have several different sets of ResNet50 weights). We need to check to make sure the pretrained weights are valid, then separately need to check to see if our trainers work with resnets, vits, etc.

We want to make sure our weights actually load in a model correctly.

Right, but the BYOL trainer is not the right place for this. We can check that the weights load in the model correctly with:

from torchgeo.models import get_model, get_model_weights, list_models
for model_name in list_models():
    for weights in get_model_weights(model_name):
        model = get_model(model_name, weights=weights)

[calebrob6]

However, I would also say we should change strict=True in the model factory type methods -- https://github.com/microsoft/torchgeo/blob/main/torchgeo/models/resnet.py#L223

[adamjstewart]

Can you make the same change to the other trainers?

[adamjstewart]

Not only does this decrease memory usage, it also shaves off 1/3 of the time our tests take to run!

It's unclear if this is a solution or if we're just kicking the can. We could find that if we double the number of models, the model tests start to fail instead of the trainer tests. But I guess we'll find out when we add all of our Landsat weights.

[isaaccorley]

If we are using dummy models, does it even make sense to test that all the model weights work in a trainer other than just testing that the weights load properly into the specified backbone?

[adamjstewart]

At a bare minimum, we need these tests for test coverage. But we also want to make sure that enums, strings, and paths all work correctly