Some callbacks for debugging and visualization

[awaelchli]

🚀 Feature

Do you have interest in adding any of these callbacks here?
https://github.com/awaelchli/pytorch-lightning-snippets
Can also make an issue for each one, just wanted to know first if there is a desire for more callbacks in PL bolts?

TrainingDataMonitor: logs histogram of what goes into training_step
ModuleDataMonitor: logs histogram of what goes into each submodule in LightningModule
BatchNormVerification: checks (heuristic) if you have a bias before a normalization layer
BatchGradientVerification: checks that your network does not mix data across batch dim.

Motivation

• Have more callbacks in PL bolts
• useful for debugging and visualization of training

Additional context

Happy to help maintain it and adapt to your needs.

[github-actions]

Hi! thanks for your contribution!, great first issue!

[nateraw]

These are 🔥 . Let's do it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TylerYep]

@awaelchli Are you still planning on merging all of these? I figured since you already wrote the code, the verifiers might as well be merged in :)

[awaelchli]

@TylerYep oh yes, thanks for the reminder.
Not sure how useful the batch norm verification is in general but the gradient check definitely, that's awesome!
I will prepare a PR