Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supporting Adding DDP Communication Hooks #6736

Merged
merged 56 commits into from
Apr 7, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
89f284d
Fix some test errors
Mar 23, 2021
80cfbff
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Mar 23, 2021
536c132
checkpoint consolidation
Mar 24, 2021
f172101
Update ddp_spawn.py
shuyingsunshine21 Mar 24, 2021
bf70e43
Update test_metric_result_integration.py
shuyingsunshine21 Mar 24, 2021
ea74906
Update test_results.py
shuyingsunshine21 Mar 24, 2021
a9aae99
Update utils.py
shuyingsunshine21 Mar 24, 2021
70fe5da
Update utils.py
shuyingsunshine21 Mar 24, 2021
0d23d75
Update test_all_gather_grad.py
shuyingsunshine21 Mar 24, 2021
ca6f98b
Update test_all_gather_grad.py
shuyingsunshine21 Mar 24, 2021
c5053da
Merge pull request #1 from shuyingsunshine21/shuyingsunshine21-checkp…
shuyingsunshine21 Mar 24, 2021
9d4a2b8
Update test_results.py
shuyingsunshine21 Mar 24, 2021
7635b4f
Revert "Update test_results.py"
shuyingsunshine21 Mar 24, 2021
d64f90c
Revert "Merge pull request #1 from shuyingsunshine21/shuyingsunshine2…
shuyingsunshine21 Mar 24, 2021
dcdcd29
Revert "Update test_all_gather_grad.py"
shuyingsunshine21 Mar 24, 2021
8651d54
Revert "Update utils.py"
shuyingsunshine21 Mar 24, 2021
15f4b9e
Revert "Update utils.py"
shuyingsunshine21 Mar 24, 2021
250d0aa
Revert "Update test_results.py"
shuyingsunshine21 Mar 24, 2021
6c095b2
Revert "Update test_metric_result_integration.py"
shuyingsunshine21 Mar 24, 2021
8222dc9
Revert "Update ddp_spawn.py"
shuyingsunshine21 Mar 24, 2021
3a9fde9
Revert "checkpoint consolidation"
shuyingsunshine21 Mar 24, 2021
7a369f4
Revert "Revert "checkpoint consolidation""
shuyingsunshine21 Mar 24, 2021
b4a0b9e
Revert "Revert "Revert "checkpoint consolidation"""
shuyingsunshine21 Mar 24, 2021
5cf1db1
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Mar 24, 2021
0ce7e05
Revert "Revert "Update ddp_spawn.py""
shuyingsunshine21 Mar 24, 2021
fe9736d
Revert "Revert "Update test_metric_result_integration.py""
shuyingsunshine21 Mar 24, 2021
c314ef6
Revert "Revert "Update test_results.py""
shuyingsunshine21 Mar 24, 2021
c3feda0
Revert "Revert "Update utils.py""
shuyingsunshine21 Mar 24, 2021
c759477
Revert "Revert "Update test_all_gather_grad.py""
shuyingsunshine21 Mar 24, 2021
7a8e540
Merge branch 'master' of https://github.com/shuyingsunshine21/pytorch…
Mar 24, 2021
ab8b849
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Mar 24, 2021
4e67db2
modify distributed environment to make test pass
Mar 24, 2021
67b6188
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Mar 25, 2021
1e41d5b
add DDP communication hook
Mar 30, 2021
6833b87
remove test related setting
Mar 30, 2021
f856d31
remove more test related setting
Mar 30, 2021
14a0a1b
fix ddp comm hook util import issue
Mar 30, 2021
8998469
comments
Mar 30, 2021
a17947b
one more fix for test_custom_plugin
Mar 30, 2021
91a945a
fix ddp spwan
Mar 30, 2021
78c6925
fix sgd
Mar 30, 2021
443f223
address comments and add tests
Mar 30, 2021
f8d0603
1. add is gpu checking 2. modify test a bit 3. formatting
Mar 31, 2021
f06285f
formatting nit
Mar 31, 2021
b607ebd
fix conda 3.7 1.7 issue for no torch.distributed.algorithms module
Mar 31, 2021
6cc9dfa
need at least 1.8.0
Apr 1, 2021
b12a16b
minor fix
Apr 1, 2021
25ccb82
modify changelog
Apr 1, 2021
35d49bc
changelog should link to PR number instead of issue number
Apr 1, 2021
dc5c55c
refine a bit on doc for register_ddp_comm_hook function, like ddp_com…
Apr 1, 2021
fb184b2
move single device checking before call register_ddp_comm_hook
Apr 1, 2021
bf44378
formatting
Apr 2, 2021
d529985
comments
Apr 5, 2021
b8105be
typo
Apr 5, 2021
e32a11d
pre-commit formatting
Apr 6, 2021
2275b45
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Apr 6, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove test related setting
  • Loading branch information
Shuying Sun committed Mar 30, 2021
commit 6833b8705091ead69c84e8d2878578ea713fbe99
6 changes: 1 addition & 5 deletions pytorch_lightning/plugins/training_type/ddp.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,7 @@
from pytorch_lightning.overrides.distributed import prepare_for_backward
from pytorch_lightning.plugins.environments.cluster_environment import ClusterEnvironment
from pytorch_lightning.plugins.training_type.parallel import ParallelPlugin
from pytorch_lightning.utilities import (
_HYDRA_AVAILABLE,
_TORCH_GREATER_EQUAL_1_7,
rank_zero_warn,
)
from pytorch_lightning.utilities import _HYDRA_AVAILABLE, _TORCH_GREATER_EQUAL_1_7, rank_zero_warn
from pytorch_lightning.utilities.distributed import rank_zero_only, ReduceOp, sync_ddp_if_available
from pytorch_lightning.utilities.exceptions import MisconfigurationException
from pytorch_lightning.utilities.seed import seed_everything
Expand Down
1 change: 1 addition & 0 deletions pytorch_lightning/utilities/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""General utilities"""
import numpy

from pytorch_lightning.utilities.apply_func import move_data_to_device # noqa: F401
from pytorch_lightning.utilities.distributed import ( # noqa: F401
Expand Down
3 changes: 0 additions & 3 deletions tests/core/test_metric_result_integration.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@
import torch.distributed as dist
import torch.multiprocessing as mp
from torchmetrics import Metric
import numpy
import os

import tests.helpers.utils as tutils
from pytorch_lightning.core.step_result import Result
Expand Down Expand Up @@ -98,7 +96,6 @@ def test_result_reduce_ddp():
tutils.set_random_master_port()

worldsize = 2
os.environ["MKL_SERVICE_FORCE_INTEL"] = "1"
mp.spawn(_ddp_test_fn, args=(worldsize, ), nprocs=worldsize)


Expand Down
3 changes: 0 additions & 3 deletions tests/core/test_results.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,6 @@
from pytorch_lightning.trainer.states import TrainerState
from tests.helpers import BoringDataModule, BoringModel
from tests.helpers.runif import RunIf
import os
import numpy


def _setup_ddp(rank, worldsize):
Expand All @@ -52,7 +50,6 @@ def _ddp_test_fn(rank, worldsize, result_cls: Result):
def test_result_reduce_ddp():
"""Make sure result logging works with DDP"""
tutils.reset_seed()
os.environ["MKL_SERVICE_FORCE_INTEL"] = "1"
tutils.set_random_master_port()

worldsize = 2
Expand Down
2 changes: 1 addition & 1 deletion tests/helpers/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ def reset_seed(seed=0):
def set_random_master_port():
reset_seed()
port = RANDOM_PORTS.pop()
os.environ['MASTER_PORT'] = "29501"
os.environ['MASTER_PORT'] = str(port)


def init_checkpoint_callback(logger):
Expand Down
2 changes: 1 addition & 1 deletion tests/metrics/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
def setup_ddp(rank, world_size):
""" Setup ddp enviroment """
os.environ["MASTER_ADDR"] = "localhost"
os.environ["MASTER_PORT"] = "29501"
os.environ["MASTER_PORT"] = "8088"

if torch.distributed.is_available() and sys.platform not in ("win32", "cygwin"):
torch.distributed.init_process_group("gloo", rank=rank, world_size=world_size)
Expand Down
2 changes: 1 addition & 1 deletion tests/utilities/test_all_gather_grad.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
def setup_ddp(rank, world_size):
""" Setup ddp enviroment """
os.environ["MASTER_ADDR"] = "localhost"
os.environ["MASTER_PORT"] = "29501"
os.environ["MASTER_PORT"] = "8088"

if torch.distributed.is_available() and sys.platform not in ("win32", "cygwin"):
torch.distributed.init_process_group("gloo", rank=rank, world_size=world_size)
Expand Down