Baal Summer Planning 2022

[Dref360]

Just like we did last year, we will hold a “Baal Summer Planning” meeting on June 15th 10h30 ET. 🎉
The goal is to come together and determine priorities for the summer.

If you would like to join to propose a project or simply to listen let me know! This will be recorded and I’ll share the recording on Slack.

This is also a great opportunity to talk to us about your issues with Baal and how we can make it better for you all.

[Dref360]

To me, the following items are my priorities:

• New Design (logo, brand)
• Organisation management (Add Sponsors, regular community meetings, move/fork experiment codebases under baal-org)
• Definitive HuggingFace tutorial based on our learnings, maybe create an HF Space to showcase this.

[Dref360]

Action item:

• Look into mkdocs/ material-ui
• Make a list of communities we want to reach out to
• Redo the CLINC-150 experiments.
• Look into getting access to GPUs (free google credit)

[GeorgePearse]

• Demos available in several GPU cloud providers (google colab, paperspace gradient) with some form of logging already functional (the research paper performance vs. dataset size out of the box), along with graphs for each training run.
• Focus on smooth/easy integration with pytorch-lightning, it has by far the greatest adoption, most research teams will not be using Flash. The no. 1 business style metric IMO should be time from package discovery to better than random selection results. Often teams have a long list of potentially very valuable things to try (composer for faster training, adoption of pytorch-lightning fullstop, latest distributed training strategy etc.) and end exploration at the first bug / hurdle. Pytorch CI may help reducing the number of package related problems.
• Focus on implementing / testing techniques that have the potential to reduce the GPU cost of experiments (careful use of fine-tuning).
• Greater availability of experimental results, either via a public Weights and Biases page, or a dashboarding tool like Superset, with a particular focus on ablation studies / hyperparameter sweeps, e.g. how much does reducing the number of runs of inference with MC dropout impact the efficacy of active learning? How overfit should a model be? etc. Make sure to then set strategy defaults to the best performing hyperparameter values.
• Some set of experiments with ensemble training and predictions because this is a really annoying experiment to run as a researcher but potentially the cheapest method as an MLE in industry because they're always training models with different hyperparameters and architectures anyway (I think this is something that companies like Tesla do a lot of)
• Number of MC iterations could potentially be determined via an accuracy tolerance.

[Dref360]

Action item:

• Undeprecate Pytorch lightning integration
• Add CI for PL
• Add some utils for easy metrics logging.
• Store log with W&B

[parmidaatg]

IMO all the above suggestions are great but we need to put a strategic focus on expanding our community. so maybe priority could be this one but then ofcourse technical side, I agree that we have a list of pain points that George mentioned and we need to cover them. I also think that it is time to have a really good HF tutorial.