Efficient ad reduced memory footprint

[MG109]

Description

I ran into an OutOfMemory exception when calculating the teacher channel mean and standard deviation while training the EfficientAD model. I fixed this by not saving the teacher outputs to a list but instead iterating the dataloader twice.

• Fixes [Task]: EfficientAD OutOfMemory exception (fix included) #1301

• TODO: Sort out pre-commit stuff (couldn't get the pre-commit hooks running)

• Maybe: Implement 'online' mean and variance calculation, see [Task]: EfficientAD OutOfMemory exception (fix included) #1301 (comment)

Changes

Describe the changes you made

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which refactors the code base)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

Ensure that you followed the following

• I have added a summary of my changes to the CHANGELOG (not for minor changes, docs and tests).
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas (If applicable)
• I have made corresponding changes to the documentation (If applicable)
• I have added tests that prove my fix is effective or that my feature works (If applicable)

[samet-akcay]

@alexriedel1, would you like to review this PR?

[alexriedel1]

@alexriedel1, would you like to review this PR?

Sure! I would prefer to only use one method without going for the try except. So just implement the twice iterating method as Standard @MG109

[alexriedel1]

looks good to me!

[samet-akcay]

Thanks for your contribution!