Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix environment variable order for global rank determination #11406

Merged
merged 9 commits into from
Feb 17, 2022
Merged

Fix environment variable order for global rank determination #11406

Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2e71684
Fix rank environment variable priority
konstantinjdobler Jan 10, 2022
8d259cf
Add comment explaining rank_keys order
konstantinjdobler Jan 12, 2022
ae805df
Merge branch 'PyTorchLightning:master' into master
konstantinjdobler Feb 3, 2022
d664486
Add changelog entry
konstantinjdobler Feb 3, 2022
9838561
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 3, 2022
2492029
Merge branch 'master' into konstantinjdobler/master
awaelchli Feb 10, 2022
2cef74c
fix merge
awaelchli Feb 10, 2022
d79ab4b
add test
awaelchli Feb 10, 2022
5e7a2d6
Merge branch 'master' into konstantinjdobler/master
awaelchli Feb 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion pytorch_lightning/utilities/distributed.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ def wrapped_fn(*args: Any, **kwargs: Any) -> Optional[Any]:

# TODO: this should be part of the cluster environment
def _get_rank() -> int:
rank_keys = ("RANK", "SLURM_PROCID", "LOCAL_RANK")
# SLURM_PROCID can be set even if Slurm is not managing the multiprocessing, therefore LOCAL_RANK needs to be checked first
rank_keys = ("RANK", "LOCAL_RANK", "SLURM_PROCID")
awaelchli marked this conversation as resolved.
Show resolved Hide resolved
awaelchli marked this conversation as resolved.
Show resolved Hide resolved
for key in rank_keys:
rank = os.environ.get(key)
if rank is not None:
Expand Down