Bias and model transparency

[dontcallmedom]

In her talk, Jutta highlighted the risks for minorities or groups whose data are underrepresented in data used for training models, and approaches to reduce the bias (e.g. the "lawnmower" approach)

@JohnRochfordUMMS 's talk highlighted that privacy concerns make that phenomenon even stronger for people with disabilities, and highlighted tools that can help identify bias in training data.

Are there well known metrics or metadata that a model provider can (and ideally, should) attach to their models to help developers assess how much and what kind of bias they might be importing when they use a given model? Are there natural fora where discussions on these metadata are expected to happen?

[anssiko]

I recently read a paper Model Cards for Model Reporting that discusses a "model card" proposal that suggests an approach where an ML model is accompanied by documentation detailing the model's limitations and performance characteristics. Perhaps this model [pun unintended] would work in the context of the web-based ML as well. In fact, it would be quite a natural fit in my view.

Here are couple of proof of concept model cards from Google's Cloud Vision API: face detection and object detection.

Edit: More examples of model cards from another project (https://mediapipe.dev/) at https://google.github.io/mediapipe/solutions/models

Adding @mmitchellai, one of the paper's authors, for comments.

[dontcallmedom]

Looking at the face detection example linked by @anssiko , it helps identifies for instance that the sample of data is very strongly skewed towards "lighter tone skins" (100K samples) where "darker tone skins" were only a 1/5th of the sample.

In any case, exposing the data sounds a really useful first step; it would be great to hear about other similar projects and if there is any convergence in this space that could be triggered by wider dev adoption (which we can assume bringing ML to the Web should induce).

[mmitchellai]

Hello! Here is the Model Card Toolkit we just released: https://ai.googleblog.com/2020/07/introducing-model-card-toolkit-for.html Here are some further examples of Model Cards, in case useful: Open AI: - https://github.com/openai/gpt-2/blob/master/model_card.md - https://github.com/openai/gpt-3/blob/master/model-card.md - https://github.com/openai/summarize-from-feedback/blob/master/model_card.md Allen AI: - https://autocat.apps.allenai.org/?uid=35706625-67f6-45e5-933b-d40c4c753a17 - https://autocat.apps.allenai.org/?uid=df78e70e-89a5-40d9-8951-36eff7642dac NVidia, "Transfer Learning Toolkit <https://ngc.nvidia.com/catalog/models>" models: - https://ngc.nvidia.com/catalog/models/nvidia:tlt_instance_segmentation - https://ngc.nvidia.com/catalog/models/nvidia:tlt_vehicletypenet - https://ngc.nvidia.com/catalog/models/nvidia:tlt_vehiclemakenet - https://ngc.nvidia.com/catalog/models/nvidia:tlt_trafficcamnet - https://ngc.nvidia.com/catalog/models/nvidia:tlt_dashcamnet - https://ngc.nvidia.com/catalog/models/nvidia:tlt_peoplenet - https://ngc.nvidia.com/catalog/models/nvidia:tlt_facedetectir - https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_detectnet_v2 - https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_classification - https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection Salesforce: - https://help.salesforce.com/articleView?id=mc_anb_einstein_messaging_insights_model_card.htm&type=5 - https://help.salesforce.com/articleView?id=mc_anb_einstein_sto_model_card.htm&type=5 NLP Open Source: https://github.com/huggingface/model_card PULSE: https://thegradient.pub/pulse-lessons/#on-best-practices-for-addressing-bias-in-ai-models Sepsis Watch: https://arxiv.org/ftp/arxiv/papers/1911/1911.08089.pdf Whatever this is: https://librecorps.github.io/resources/ai/model-card/ HTH!

…

On Thu, Sep 24, 2020 at 2:39 AM Dominique Hazael-Massieux < ***@***.***> wrote: Looking at the face detection <https://modelcards.withgoogle.com/face-detection#limitations> example linked by @anssiko <https://github.com/anssiko> , it helps identifies for instance that the sample of data is very strongly skewed towards "lighter tone skins" (100K samples) where "darker tone skins" were only a 1/5th of the sample. In any case, exposing the data sounds a really useful first step; it would be great to hear about other similar projects and if there is any convergence in this space that could be triggered by wider dev adoption (which we can assume bringing ML to the Web should induce). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#108 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAL7FLF4OD7QTGNHVLSOTUTSHMHTLANCNFSM4RWVXZDA> .

[Jutta-Inclusive]

Knowing the limitations and potential bias of the training data is a great step in the right direction. Also knowing what proxy data was used to fill gaps would be helpful.
Some of my concerns go deeper. Even with full proportional representation and the teasing out of systemic bias, decisions based on population data will always decide against the minority. This is a "turtles all the way down" issue in that even personalized systems will work better for the majority than the minority. How do we address this bias that preexists and is then amplified and automated in ML?

Some of the groups looking at this include:
The Montreal AI Ethics group: https://montrealethics.ai/
and our We Count Project: https://wecount.inclusivedesign.ca/

Jutta

[toreini]

Hi,

This is an interesting topic; model transparency and explainability are not easy. Is there any thought on how one can challenge the output of the ML model? I get the impression that Model cards tend to make the whole model transparent. Can it explain the decision made for a single data point (for instance, she/he wants to know why this decision has been made for her/his own case)?

Thanks,
Ehsan