powersgd

[titusnicolae]

Powersgd paper shows promising scaling for distributed training.
https://arxiv.org/abs/1905.13727
I'm interested in porting this to pytorch-lightning, do you think it's a good idea?
Thanks

[jeffling]

@titusnicolae I'm a big fan of this. I'm not sure if it needs to be a framework level feature, or if we should find a way to make it easy to plug in. But I would love to see a lightning-compatible implementation of this, whether in a PR or as a separate plugin.

[williamFalcon]

@titusnicolae @soumith this might be better in PyTorch?

[Borda]

@titusnicolae @jeffling any update here?

[jeffling]

@titusnicolae any luck on porting this over?

If it could be done in a callback, it should be. otherwise, I think if there is enough proof / demand behind this it would make sense to be in the framework.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[Borda]

@edenlightning is this in timeline?

[tchaton]

Hey @jeffling,

Any updates ? You could make a PR to add it to Lightning directly.

Best,
T.C

[tchaton]

Dear @titusnicolae,

PyTorch currently support PowerSGD and it can easily be added by registering builtin DDP Communitation hooks.

Do you want to give it a try ?

Best,
T.C

[titusnicolae]

thanks Thomas! I'll give it a try someday, I'll be busy the following month

…

On Tue, Mar 9, 2021, 14:38 thomas chaton ***@***.***> wrote: Dear @titusnicolae <https://github.com/titusnicolae>, PyTorch currently support PowerSGD and it can easily be added by registering builtin DDP Communitation hooks. Do you want to give it a try ? Best, T.C — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#643 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD7EP2G6W67FNAFKVUGCDDTCYJELANCNFSM4J6A67JQ> .

[shuyingsunshine21]

looks like #6727 could cover this scenario :)

[shuyingsunshine21]

@titusnicolae , with #6736, this should be supported