Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for ObjectDetection #758

Merged
merged 10 commits into from
Oct 4, 2022
Merged

Conversation

ashnair1
Copy link
Collaborator

@ashnair1 ashnair1 commented Sep 5, 2022

Adds a new ObjectDetectionTask which uses torchvision's Faster-RCNN under the hood.

Closes #442

@github-actions github-actions bot added datamodules PyTorch Lightning datamodules datasets Geospatial or benchmark datasets testing Continuous integration testing trainers PyTorch Lightning trainers labels Sep 5, 2022
@ashnair1 ashnair1 force-pushed the detection-task branch 2 times, most recently from c6a6a87 to ee680fe Compare September 5, 2022 12:15
@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 5, 2022

So the tests are failing due to warnings caused by the new Multi-Weight API torchvision introduced in 0.13.

If we want to continue supporting the minimum torchvision 0.10.0, I could just ignore the warnings. Otherwise, I can update detection.py to use the new API and bump minimum torchvision version to 0.13.

What do you guys think?

@adamjstewart
Copy link
Collaborator

This is great, thanks so much!

For torchvision dependency version, see what I did in #631 for the other trainers. It's possible to check the version of torchvision and use different syntax for each version. It's kinda ugly and I don't like it, but I'm not sure if it's wise to just drop support for all older torchvision versions. Open to feedback on this though.

For the torchmetrics dependency version, what was the reason for the version bump? It seems like MeanAveragePrecision was introduced in 0.7.0 (renamed from MAP) according to the changelog.

@adamjstewart adamjstewart added this to the 0.4.0 milestone Sep 5, 2022
@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 5, 2022

For torchvision dependency version, see what I did in #631 for the other trainers. It's possible to check the version of torchvision and use different syntax for each version. It's kinda ugly and I don't like it, but I'm not sure if it's wise to just drop support for all older torchvision versions. Open to feedback on this though.

Saw this snippet.

if parse(torchvision.__version__) >= parse("0.12"):
    if backbone_pretrained:
        kwargs = {
            "weights": getattr(
                torchvision.models, f"ResNet{backbone[6:]}_Weights"
            ).DEFAULT
        }
    else:
        kwargs = {"weights": None}
  else:
    kwargs = {"pretrained": backbone_pretrained}

This can work. Since the number of backbones supported also include the ResNext and Wide_Resnet variants, I opted to create a mapping between backbones and weights. Perhaps this could be moved to a common location in a later PR.

Agree with not dropping support. We only need to drop a version if it breaks something or if it requires really ugly code to handle it. Think we're good for now.

For the torchmetrics dependency version, what was the reason for the version bump? It seems like MeanAveragePrecision was introduced in 0.7.0 (renamed from MAP) according to the changelog.

You're right. Think it was just the filename that was changed (map.py -> mean_ap.py). Reverted that commit.

@adamjstewart
Copy link
Collaborator

Another option would be to change the pretrained kwarg on our end so that instead of taking a boolean it takes whatever torchvision wants. Then it's up to the user to decide which pretrained model to use. Thoughts?

@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 5, 2022

Another option would be to change the pretrained kwarg on our end so that instead of taking a boolean it takes whatever torchvision wants. Then it's up to the user to decide which pretrained model to use. Thoughts?

Sure, that could work. That way the user would be able to specify the weights to be used and if they leave it unspecified, we can just use the default weights. But specifying weights is only possible for torchvision > 0.12. For lower versions we will need to pass pretrained=True regardless of the weights specified by the user.

@adamjstewart
Copy link
Collaborator

Hmm, but then it becomes difficult to store this configuration in a YAML file...

@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 5, 2022

Right. The YAML would only be informative if you're using torchvision > 0.12

@adamjstewart
Copy link
Collaborator

But even if the YAML is torchvision > 0.12 specific, can you store objects in YAML or only strings/floats/bools? We could store a string and exec it but that feels like a security risk...

@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 5, 2022

The weights in the YAML can be string. We could just specify it like this.

experiment:
  task: "nasa_marine_debris"
  name: "nasa_marine_debris_test"
  module:
    detection_model: "faster-rcnn"
    backbone: "resnet50"
+   pretrained: "ResNet50_Weights.IMAGENET1K_V1"
    num_classes: 2
    learning_rate: 1.2e-4
    learning_rate_schedule_patience: 6
    verbose: false
  datamodule:
    root_dir: "data/nasamr/nasa_marine_debris"
    batch_size: 4
    num_workers: 56
    val_split_pct: 0.2

The pretrained arg can then be passed into the weights arg (once we pass it through torchvision's get_weight function)

@adamjstewart
Copy link
Collaborator

Ah, we could use torchvision.prototype.models.get_weight to convert the string to the enum. But yeah, it would be a bit frustrating for the YAML to only work properly with torchvision 0.12.

Let's keep it like this for now but strongly consider dropping support for older torchvision at some point.

@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 5, 2022

The pretrained arg will officially be dropped in version 0.15. Perhaps that would be a good time to drop old versions and peg minimum to 0.13.

@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 6, 2022

Coverage drop is mainly caused by the code required for supporting minimum versions.

On that note, has anybody looked into combining the coverage reports for normal tests and minimal tests via coverage combine before uploading to codecov?

@adamjstewart
Copy link
Collaborator

We run tests on both the latest version of dependencies and the minimum version of dependencies. Both tests generate coverage reports, and should automatically be merged when uploaded to codecov. This works on all of our other PRs, not sure why coverage is dropping for you.

@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 6, 2022

This is weird. The coverage drop seems to be caused by unrelated files. Also a warning is shown stating there are unmerged base commits. Any ideas?

image

@adamjstewart
Copy link
Collaborator

The coverage drop seems to be caused by unrelated files.

This usually means that coverage has increased in main since this branch was created. Usually, rebase or merge commits solve this. Wish codecov was smarter about this...

@ashnair1
Copy link
Collaborator Author

ashnair1 commented Sep 6, 2022

Since I've opened this PR, there have been a total of 4 commits merged to main. The two tutorial commits, one related to Sentinel2 and the torchvision constraint fix so it's unlikely a coverage increase is the reason.

I wonder if this has to do with me rebasing and squashing commits in this branch.

@adamjstewart
Copy link
Collaborator

Yep, I have no idea what's going on here. It's just this PR, right? Probably best to reach out to codecov and see if this is a bug in codecov.

I'm focusing on 0.3.1 stuff at the moment, but if this is still broken after 0.3.1 is finished I can try to report it to codecov myself.

@isaaccorley
Copy link
Collaborator

I went down the torchvision.detection rabbit hole awhile back. I think my only complaint with it was that they don't support negative sample images that don't have any objects in them during training. Do you know if this has changed? I need to catch up on all the releases.

@isaaccorley
Copy link
Collaborator

Also I believe they added support for constructing a resnet fpn backbone from a pretrained torchvision resnet model. @calebrob6 and @anthonymlortiz, this might be of interest because we could add support for pretrained SSL resnet backbones to this task.

@ashnair1 ashnair1 requested a review from adamjstewart October 3, 2022 08:10
adamjstewart
adamjstewart previously approved these changes Oct 3, 2022
@adamjstewart
Copy link
Collaborator

@isaaccorley does this need a predict step?

adamjstewart
adamjstewart previously approved these changes Oct 3, 2022
@isaaccorley
Copy link
Collaborator

@isaaccorley does this need a predict step?

Ideally yes, we can have a predict step that returns the predicted boxes and class labels per sample.

@adamjstewart
Copy link
Collaborator

Let's add that here before we forget, I want to make sure all of our trainers support the same things.

@isaaccorley
Copy link
Collaborator

@ashnair1 Let me know if you want some help on it. I can walk you through the predict_steps I made for the classifiers.

@ashnair1
Copy link
Collaborator Author

ashnair1 commented Oct 4, 2022

Added predict_step and corresponding test.

@adamjstewart adamjstewart merged commit a7dbd9c into microsoft:main Oct 4, 2022
@adamjstewart adamjstewart mentioned this pull request Oct 4, 2022
6 tasks
@ashnair1 ashnair1 deleted the detection-task branch October 4, 2022 16:37
yichiac pushed a commit to yichiac/torchgeo that referenced this pull request Apr 29, 2023
* Prepare NasaMarineDebris dataset & datamodule

* Filter out invalid boxes
* Add label key to batch
* Add plot function

* Add detection task

* Add tests

* Fix conf arg

* Add test for non pretrained backbones

* Coverage for when datamodule has no plot fn

* Separate out tests

* self.forward(x) -> self(x)

* Add predict_step

* list -> List
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datamodules PyTorch Lightning datamodules datasets Geospatial or benchmark datasets testing Continuous integration testing trainers PyTorch Lightning trainers
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request] Object Detection Trainer
4 participants