Replace adam(b1=0) by rmsprop for schedule_free

[copybara-service]

Replace adam(b1=0) by rmsprop for schedule_free

[vroulet]

Closes #1006

[nasyxx]

Is it possible to rename the schedule_free_adamw to schedule_free_rms or mention in the document that it is actually using rms. Although when beta1 is 0, they are almost the same.

[vroulet]

1. What do you mean by they are "almost" the same? There is a test that checked that they match. If you found non-negligible discrepancies, please let us know: file a bug with a minimal reproducing example.
2. Provided that there is no difference between the wrapped version and the shortuct version (see tests), the implementation matches the implementation of schedule_free with adamw as intended by the authors so we won't rename it.
3. I can add indeed a note that the implementation is using rmsprop for efficiency reasons, that's a good point. Could you open an issue on that, so that I don't forget? You may also directly create the PR that adds this information on the docstring.

[nasyxx]

1. What do you mean by they are "almost" the same? There is a test that checked that they match. If you found non-negligible discrepancies, please let us know: file a bug with a minimal reproducing example.

The tests are matched. I meant when b1 is 0, Adam's and RMS's proposals are almost the same.

3. I can add indeed a note that the implementation is using rmsprop for efficiency reasons, that's a good point. Could you open an issue on that, so that I don't forget? You may also directly create the PR that adds this information on the docstring.

Yes, I just checked another issue and found that it reduces memory usage.

[vroulet]

I meant when b1 is 0, Adam's and RMS's proposals are almost the same.

Are they "exactly" or "almost" the same? To me they are exactly the same. If not, please let us know what are exactly the differences.

[nasyxx]

Are they "exactly" or "almost" the same? To me they are exactly the same. If not, please let us know what are exactly the differences.

The difference between Adam and RMSprop is that Adam has bias. However, yes, with bias_correction=True in RMS, they should be exactly the same.