Question about overfitting mitigation in LIMO training

[TweedBeetle]

Hi! I'm trying to replicate LIMO's training setup on a slightly altered version of your dataset. I've noticed significant overfitting in my training runs, but unlike your results, I'm not seeing good performance on out-of-distribution tasks.

I'm particularly interested in:

1. Did you implement any specific techniques to mitigate overfitting given the very small dataset size (817 samples)?
2. Did you find that overfitting on the training set was actually not detrimental to OOD task performance in your case?
3. Was your choice of learning rate part of the overfitting mitigation strategy?

Would greatly appreciate any insights into how you approached this challenge in the original implementation.

Thanks!