Potential Exploding Gradients Problem?

[navajain19]

Hi @sophiajw! I was wondered if you have ever encountered the following problem when training your model. Based on what I've found online, it appears to be an exploding gradients problem, and I have been experimenting with different learning rate schedules. Thanks!

[sophiajw]

Hey navajain19,
did you use the latest version of the repo? Indeed, I also encountered this before and it was as an exploding gradient problem. In a previous commit, another user corrected a typo in the gaussian weight init (0.02 instead of 0.2), which led to the same mistake.

[navajain19]

Hi - thanks for letting me know. I gaussian weight init is indeed 0.02, but I am just experimenting with another learning rate scheduler, so it could be that the LR got too high. Thanks for confirming this for me.