Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AdaBelief optimizer #2548

Merged
merged 21 commits into from
Aug 25, 2021
Merged

Conversation

juntang-zhuang
Copy link
Contributor

@juntang-zhuang juntang-zhuang commented Aug 22, 2021

Description

Add AdaBelief in tensorflow_addons, which adapts stepsize according to "belief" in gradient, and achieves good generalization, fast convergence and training stability.

Reference: AdaBelief optimizer: adapting stepsizes by the belief in observed gradients (Juntang Zhuang et al. NeurIPS 2020).

Results: 77.25 (AdaBelief) 77.1 (SGD)

Type of change

Checklist

  • I've properly formatted my code according to the guidelines
    • By running Black + Flake8
    • By running pre-commit hooks
  • This PR addresses an already submitted issue for TensorFlow Addons
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • This PR contains modifications to C++ custom-ops

How Has This Been Tested?

By running a ResNet50 model from google/tensorflow_models benchmark

If you're adding a bugfix or new feature please describe the tests that you ran to verify your changes:
*

@AakashKumarNain
Copy link
Member

@juntang-zhuang please format the code, and make sure your flake8 and black versions are same as in tf-addons

@AakashKumarNain AakashKumarNain merged commit a69c12a into tensorflow:master Aug 25, 2021
@AakashKumarNain
Copy link
Member

Thank you for the contribution. This is a good addition to addons

@juntang-zhuang
Copy link
Contributor Author

Thanks for your help! @AakashKumarNain

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants