Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bugfix] Fix GPT script #36

Merged
merged 1 commit into from
Feb 1, 2023
Merged

[Bugfix] Fix GPT script #36

merged 1 commit into from
Feb 1, 2023

Conversation

szhengac
Copy link
Contributor

@szhengac szhengac commented Feb 1, 2023

Description

head sch needs allreduce on input grad. Now the loss curve is consistent with 2D and ZeRO 3.

Checklist

  • PR's title starts with a category (e.g. [Bugfix], [Model], [Tutorial], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

comaniac
comaniac approved these changes Feb 1, 2023
View reviewed changes
Copy link
Contributor

@comaniac comaniac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@szhengac szhengac merged commit 997dd49 into awslabs:main Feb 1, 2023
@szhengac szhengac deleted the gpt-fix branch February 1, 2023 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@comaniac comaniac comaniac approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@szhengac @comaniac