Document default trainer metrics

[jdilger]

Expanded description of averaging reduction method for Segmentation metrics. Let me know if you'd like me to change anything, or if I should add the details on macro accuracy.

Closes #1874

[adamjstewart]

Looks fantastic, maybe @robmarkcole can review?

Should we do this for all of our LightningModules? Basically just document the default metrics we use.

[robmarkcole]

I feel this is a particularly nuanced issue so have expanded slightly:

"""
Sets up segmentation accuracy metrics:

- Multiclass Pixel Accuracy: Ratio of correctly classified pixels.
- Multiclass Jaccard Index (IoU): Per-pixel overlap between predicted and
  actual segments.

Uses 'micro' averaging, aggregating pixel predictions across classes. Each
pixel is weighted equally, favoring classes with more pixels. This may skew
performance metrics towards majority classes due to class imbalance.

Note:
- 'Micro' averaging suits overall performance evaluation but may not reflect
  minority class accuracy.
- 'Macro' averaging, not used here, gives equal weight to each class, useful
  for balanced performance assessment across imbalanced classes.
"""

[adamjstewart]

@jdilger do you think you can produce a similar level of detailed descriptions for the accuracy metrics used in other trainers? If not maybe @robmarkcole and I can help.

[jdilger]

Thanks @robmarkcole, the more nuanced version looks great. @adamjstewart Sure thing. I can take a first pass at it.

[adamjstewart]

We're likely going to release v0.5.2 tomorrow or Saturday, so if you can add these today that would be great. If not, we may merge this anyway and then we can add more details for more trainers in a follow-up PR. Thanks again for your help!

[robmarkcole]

LGTM. Only a minor suggestion for consistency

Sets up segmentation accuracy metrics >> Initialize the metrics

[jdilger]

I have time to work on this today, but I went ahead and pushed the suggested change from robmarkcole.

I noticed that the metrics setup for the classification and regression trainers are set up with dicts while segmentation and detection use lists. I'm not sure it needs to be addressed now, but defining some standard names might be helpful. For instance, in the segmentation trainer it's using [MulticlassAccuracy(..., average="micro",)] but in the classification trainer it's setup as {'OverallAccuracy':MulticlassAccuracy(..., average="micro",)}

@microsoft-github-policy-service agree

[adamjstewart]

There are some formatting issues with lists that are causing the documentation to fail to build. We should be using the https://sublime-and-sphinx-guide.readthedocs.io/en/latest/lists.html#unordered-lists format and indenting lines that stretch across multiple lines. Also, we could be using a note: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/notes_warnings.html.

[adamjstewart]

Also I'm happy to handle the formatting stuff myself if you want to focus on adding something to the other trainers. For better or for worse, I know most of the quirks of Sphinx/rST now.

[jdilger]

@adamjstewart I think I figured it out, but do double-check 😄 . If there are still issues I'm happy to let you take it over. I don't have too much experience with Sphinx.

Side note: In contributing.rst to install the docs is written as pip install .[docs] which didn't work for me. I had to use pip install '.[docs]'. I don't know if it's a typo or related to the OS I'm using (Mac).

[adamjstewart]

Hmm, quotes shouldn't be required for pip. What error message did you get? What shell are you using, bash?

[jdilger]

Ah, you're 100% right. I'm using zsh which gave the error zsh: no matches found: .[docs]

[adamjstewart]

Formatting looks good now. If we can do this for all other trainers by tomorrow, great. If not, we'll still merge and add more details after the release.

[jdilger]

Added a first pass for the other configuration metrics. The formatting should be correct but please double-check the content :).

[adamjstewart]

Ah, you're 100% right. I'm using zsh which gave the error zsh: no matches found: .[docs]

It looks like zsh uses square brackets for pattern matching. Feel free to open a PR to add quotes around all use of square brackets in Installation and Contributing.

[adamjstewart]

@robmarkcole last chance to review

[robmarkcole]

For the classification metrics, are they per pixel or per image?

[adamjstewart]

Classification is per-image. I'm not actually sure if micro and macro are different under that context.

[robmarkcole]

I think we expect them to be different for imbalanced datasets. The metrics will not compute at the pixel level but at the image level - the terms total true positives, false negatives, and false positives should be used for classification

[adamjstewart]

Want to squeeze this in to the next release, but feel free to open another PR to update these docs or change the default metrics we use after the release!