some accelerator enhancements

[ZhiyuanChen]

Hi,

My main code base have been switched to be based on accelerate for a while, it works grate! Thank you guys so much!

There are a few things I think it's a common need.

The first is .distsibuted attributes, which is basically .num_processes > 1. I mean, it's not a lot of works but .... I'm an extremely lazy person.

def distsibuted(self):
    """
    Runs in distributed mode
    """
    return self.num_processes > 1

Besides, these two decorators are often used in my code when saving models and downloading datasets, I think it could be useful.

def on_main_process(func):
    """
    Run func on main process
    """
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        if self.is_main_process or not self.distributed:
            return func(self, *args, **kwargs)
    return wrapper


def on_local_main_process(func):
    """
    Run func on local main process
    """
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        if self.is_local_main_process or not self.distributed:
            return func(self, *args, **kwargs)
    return wrapper

[muellerzr]

Nice idea! @sgugger WDYT of instead having this wrap around AcceleratorState directly, rather than having this be attached to Accelerator?

[sgugger]

I don't mind having wrappers on the Accelerator for those. As long as the naming is clear, it's not a bad thing to have multiple properties/methods/context managers for different use cases :-)

[ZhiyuanChen]

I do think distsibuted should be in AcceleratorState, but for the decorators, I think it's better to put it in the Accelerator

In this case, should distsibuted be a property or a variable?

Property gives better support in elastic training, variable is more consitent with existing design.

[sgugger]

There is no way to set it and make it work (you can't transform the training in distributed mode suddenly), so it should be a property.

[ZhiyuanChen]

There is no way to set it and make it work (you can't transform the training in distributed mode suddenly), so it should be a property.

Ohh, by property I meant the @property decorated function, and by variable I meant a value. So I assume variable is better.

Though as PyTorch is supporting elastic training, maybe other values (e.g. num_processes) should be set to a property?

[sgugger]

I understood properly, and I still stand by the fact it should be a property, decorated by @property and not a variable.

[ZhiyuanChen]

Should we makee other variables (e.g. num_processes) property too?

[muellerzr]

@ZhiyuanChen I'd say it couldn't hurt I think!

[ZhiyuanChen]

Hi sorry for this late reply.... but do you mean if changing doesn't hurt or not changing doesn't hurt...@muellerzr

[muellerzr]

Changing it! 😄

[ZhiyuanChen]

Changing it! 😄

Haha will do

[ZhiyuanChen]

I believe everything has been merged now, so I'm gonna close this issue.

Thanks again for this grate work!