Cannot reproduce results for the experiment 1 of the paper

[michalzajac-ml]

Hello,

Thanks for sharing the code.
However, I've had trouble reproducing the results, starting with experiment 1.
I've set up a fresh Python 3.7 environment and installed dependencies with bash setup_and_install.sh. Then I've run bash replicate_experiment_1.sh and here are the results I've got:

• rotated mnist, naive: 87.0, stable: 86.8 (average accuracy, averaged over 3 runs). According to the paper, stable should be > 90 (Figure 3c)
• permuted mnist, naive: 77.7, stable: 61.7. According to the paper, stable should be > 90.

Any hints for reproducing the results would be very helpful.

[imirzadeh]

Dear @zajaczajac,

Thank you for the feedback. I think I made a typo error for dropout rate. It should have been 0.25 rather than 0.5. I hope this helps. Additionally, you can try different hyper-parameters and even get better results. Here are some runs:

Sorry for the inconvenience. I'm really busy this summer but I intend to provide completely reproducible results in codalab. In addition, I have recorded my experiments online (using comet.ml) and I can share some specific results with you. But the code is not as clean as this repository.

Regards,
Iman

[michalzajac-ml]

Dear Iman,

Thanks for your quick response! Changing dropout value indeed helps for rotated mnist experiment. However, it does not fully work for permuted mnist - performance is now at 81.15. Do you have some additional insights here?
Also, if you can share the results from comet.ml, that would be helpful. Can you please e-mail the link to me? (my e-mail should be visible at my github profile)

Thanks,
Michał

[imirzadeh]

You are welcome :)

Sure. I just sent you an email.
I close this issue since I believe it is resolved for now.