[Feature] Add context manager to allow main process first.

[Guillem96]

This feature is useful when downloading or processing data.

Example

accelerator = Accelerator() 
cache_fname = Path("cache.txt") 

with accelerator.main_process_first():  ## or accelerator.local_main_process_first()
    if cache_fname.exists(): 
        data = cache_fname.read_text() # Load from cache 
    else:
         data = "data" # Time consuming operation 
         cache_fname.write_text(data)  # Cache the data

This syntaxis is much more concise and contributes with more semantic than the older one:

if not accelerator.is_main_process:  # or is_local_main_process
    accelerator.wait_for_everybody()

if cache_fname.exists(): 
    data = cache_fname.read_text() # Load from cache 
else:
     data = "data" # Time consuming operation 
     cache_fname.write_text(data)  # Cache the data

if accelerator.is_main_process:  # or is_local_main_process
    accelerator.wait_for_everybody()

[sgugger]

Very nice addition, thanks a lot!

[sgugger]

Small nit: can we use is_main for consistency here (as well as at line 211 below)?

[Guillem96]

Yes sure!

[Guillem96]

Addressed the request.

[sgugger]

Thanks!