[BUG] terminate called after throwing an instance of 'std::bad_alloc'

[shisi-cc]

Describe the bug
When I run the code rlhf with trlx using deepspeed with two nodes, I met a strange problem "terminate called after throwing an instance of 'std::bad_alloc'". Memory and video memory are far from used up. Running on a separate machine works fine, but errors occur when two nodes are used. This problem occurs when I run with a docker container, but not when I don't use a container. In addition, I use anaconda environment.

ds_report output
(trlx_env) root@9a3cd98dd64f:/data/work/trlx_rlhf/sft# deepspeed --hostfile=../../hostfile train_gptj_summarize.py
[2023-04-03 10:49:33,397] [INFO] [runner.py:454:main] Using IP address of 10.0.128.5 for node localhost
[2023-04-03 10:49:33,398] [INFO] [multinode_runner.py:65:get_cmd] Running on the following workers: localhost,deepspeed-18
[2023-04-03 10:49:33,398] [INFO] [runner.py:548:main] cmd = pdsh -S -f 1024 -w localhost,deepspeed-18 export PYTHONPATH=/data/work/trlx_rlhf/sft; cd /data/work/trlx_rlhf/sft; /root/mambaforge/envs/trlx_env/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF0sICJkZWVwc3BlZWQtMTgiOiBbMF19 --node_rank=%n --master_addr=10.0.128.5 --master_port=29500 train_gptj_summarize.py
deepspeed-18: [2023-04-03 10:49:35,192] [INFO] [launch.py:142:main] WORLD INFO DICT: {'localhost': [0], 'deepspeed-18': [0]}
deepspeed-18: [2023-04-03 10:49:35,192] [INFO] [launch.py:148:main] nnodes=2, num_local_procs=1, node_rank=1
deepspeed-18: [2023-04-03 10:49:35,192] [INFO] [launch.py:161:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0], 'deepspeed-18': [1]})
deepspeed-18: [2023-04-03 10:49:35,192] [INFO] [launch.py:162:main] dist_world_size=2
deepspeed-18: [2023-04-03 10:49:35,192] [INFO] [launch.py:164:main] Setting CUDA_VISIBLE_DEVICES=0
localhost: [2023-04-03 10:49:35,240] [INFO] [launch.py:142:main] WORLD INFO DICT: {'localhost': [0], 'deepspeed-18': [0]}
localhost: [2023-04-03 10:49:35,240] [INFO] [launch.py:148:main] nnodes=2, num_local_procs=1, node_rank=0
localhost: [2023-04-03 10:49:35,240] [INFO] [launch.py:161:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0], 'deepspeed-18': [1]})
localhost: [2023-04-03 10:49:35,240] [INFO] [launch.py:162:main] dist_world_size=2
localhost: [2023-04-03 10:49:35,240] [INFO] [launch.py:164:main] Setting CUDA_VISIBLE_DEVICES=0
deepspeed-18: Tokenizer loaded!
localhost: Tokenizer loaded!
deepspeed-18: Model loaded!
deepspeed-18: Downloading and preparing dataset parquet/openai_summarize_tldr to /root/.cache/huggingface/datasets/parquet/openai_summarize_tldr-bed27f7b4c8f201f/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
localhost: Model loaded!
localhost: Found cached dataset parquet (/root/.cache/huggingface/datasets/parquet/openai_summarize_tldr-0275f923823d6c0b/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
localhost: Found cached dataset parquet (/root/.cache/huggingface/datasets/parquet/openai_summarize_tldr-0275f923823d6c0b/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
localhost: Dataset loaded!
localhost: [2023-04-03 10:50:46,311] [INFO] [comm.py:657:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Downloading data files: 100%|██████████| 3/3 [00:00<00:00, 10941.66it/s]
Extracting data files: 100%|██████████| 3/3 [00:00<00:00, 1896.44it/s]
deepspeed-18: Dataset parquet downloaded and prepared to /root/.cache/huggingface/datasets/parquet/openai_summarize_tldr-bed27f7b4c8f201f/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec. Subsequent calls will reuse this data.
Found cached dataset parquet (/root/.cache/huggingface/datasets/parquet/openai_summarize_tldr-bed27f7b4c8f201f/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
deepspeed-18: Dataset loaded!
Downloading builder script: 100%|██████████| 6.27k/6.27k [00:00<00:00, 26.9kB/s]
deepspeed-18: terminate called after throwing an instance of 'std::bad_alloc'
deepspeed-18: what(): std::bad_alloc
deepspeed-18: [2023-04-03 10:51:15,307] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 1231493
deepspeed-18: [2023-04-03 10:51:15,308] [ERROR] [launch.py:324:sigkill_handler] ['/root/mambaforge/envs/trlx_env/bin/python', '-u', 'train_gptj_summarize.py', '--local_rank=0'] exits with return code = -6
pdsh@9a3cd98dd64f: deepspeed-18: ssh exited with exit code 250

Hostfile
localhost slots=1
deepspeed-18 slots=1

Launcher context
deepspeed --hostfile=../../hostfile train_gptj_summarize.py

Docker context
This problem occurs when I run with a docker container, but not when I don't use a container.

[tjruwase]

Please see recent DeepSpeed Chat release #3186
https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat

[tjruwase]

@shisi-cc, did the link above help? Can this issue be closed? Thanks!

[Ancrilin]

Hi, I have encountered the same issue. I created a docker container on two different machines and ran DeepSpeed-Chat/training/step1_supervised_finetuning/muti_node/run_66b.sh, but I encountered the same error.

hostfile
node1 slots=8
node2 slots=8

ds_report_output
[2023-04-26 06:34:22,975] [INFO] [multinode_runner.py:70:get_cmd] Running on the following workers: node1,node2
[2023-04-26 06:34:22,975] [INFO] [runner.py:540:main] cmd = pdsh -S -f 1024 -w node1,node2 export NCCL_VERSION=2.12.10-1; export PYTHONPATH=/workspace/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning; cd /workspace/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning; /opt/conda/bin/python -u -m deepspeed.launcher.launch --world_info=eyJub2RlMSI6IFszLCA1XSwgIm5vZGUyIjogWzAsIDFdfQ== --node_rank=%n --master_addr=10.176.50.36 --master_port=32783 main.py --data_path 'Dahoas/rm-static' --data_split '2,4,4' --model_name_or_path '/workspace/models/opt-1.3b' --per_device_train_batch_size '1' --per_device_eval_batch_size '1' --max_seq_len '512' --learning_rate '9.65e-6' --weight_decay '0.1' --num_train_epochs '2' --gradient_accumulation_steps '1' --lr_scheduler_type 'cosine' --num_warmup_steps '0' --seed '1234' --zero_stage '3' --deepspeed --output_dir './output'
node1: [2023-04-26 06:34:26,299] [INFO] [launch.py:222:main] 0 NCCL_VERSION=2.12.10-1
node1: [2023-04-26 06:34:26,299] [INFO] [launch.py:229:main] WORLD INFO DICT: {'node1': [3, 5], 'node2': [0, 1]}
node1: [2023-04-26 06:34:26,299] [INFO] [launch.py:235:main] nnodes=2, num_local_procs=2, node_rank=0
node1: [2023-04-26 06:34:26,299] [INFO] [launch.py:246:main] global_rank_mapping=defaultdict(<class 'list'>, {'node1': [0, 1], 'node2': [2, 3]})
node1: [2023-04-26 06:34:26,299] [INFO] [launch.py:247:main] dist_world_size=4
node1: [2023-04-26 06:34:26,299] [INFO] [launch.py:249:main] Setting CUDA_VISIBLE_DEVICES=3,5
node2: [2023-04-26 06:34:28,198] [INFO] [launch.py:222:main] 1 NCCL_VERSION=2.12.10-1
node2: [2023-04-26 06:34:28,198] [INFO] [launch.py:229:main] WORLD INFO DICT: {'node1': [3, 5], 'node2': [0, 1]}
node2: [2023-04-26 06:34:28,198] [INFO] [launch.py:235:main] nnodes=2, num_local_procs=2, node_rank=1
node2: [2023-04-26 06:34:28,198] [INFO] [launch.py:246:main] global_rank_mapping=defaultdict(<class 'list'>, {'node1': [0, 1], 'node2': [2, 3]})
node2: [2023-04-26 06:34:28,199] [INFO] [launch.py:247:main] dist_world_size=4
node2: [2023-04-26 06:34:28,199] [INFO] [launch.py:249:main] Setting CUDA_VISIBLE_DEVICES=0,1
node1: [2023-04-26 06:34:30,128] [INFO] [comm.py:586:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
node1: Traceback (most recent call last):
node1: File "/workspace/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py", line 343, in
node1: main()
node1: File "/workspace/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py", line 185, in main
node1: deepspeed.init_distributed()
node1: File "/opt/conda/lib/python3.9/site-packages/deepspeed/comm/comm.py", line 588, in init_distributed
node1: cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
node1: File "/opt/conda/lib/python3.9/site-packages/deepspeed/comm/torch.py", line 32, in init
node1: self.init_process_group(backend, timeout, init_method, rank, world_size)
node1: File "/opt/conda/lib/python3.9/site-packages/deepspeed/comm/torch.py", line 58, in init_process_group
node1: torch.distributed.init_process_group(backend,
node1: File "/opt/conda/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 754, in init_process_group
node1: store, rank, world_size = next(rendezvous_iterator)
node1: File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rendezvous.py", line 246, in _env_rendezvous_handler
node1: store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout)
node1: File "/opt/conda/lib/python3.9/site-packages/torch/distributed/rendezvous.py", line 177, in _create_c10d_store
node1: return TCPStore(
node1: RuntimeError: Stop_waiting response is expected
node1: terminate called after throwing an instance of 'std::bad_alloc'
node1: what(): std::bad_alloc
node1: [2023-04-26 06:34:31,325] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 1846
node1: [2023-04-26 06:34:31,328] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 1847
node1: [2023-04-26 06:34:31,328] [ERROR] [launch.py:434:sigkill_handler] ['/opt/conda/bin/python', '-u', 'main.py', '--local_rank=1', '--data_path', 'Dahoas/rm-static', '--data_split', '2,4,4', '--model_name_or_path', '/workspace/models/opt-1.3b', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--max_seq_len', '512', '--learning_rate', '9.65e-6', '--weight_decay', '0.1', '--num_train_epochs', '2', '--gradient_accumulation_steps', '1', '--lr_scheduler_type', 'cosine', '--num_warmup_steps', '0', '--seed', '1234', '--zero_stage', '3', '--deepspeed', '--output_dir', './output'] exits with return code = -6
pdsh@4e68f64d7185: node1: ssh exited with exit code 250
node2: terminate called after throwing an instance of 'std::bad_alloc'
node2: what(): std::bad_alloc
node2: terminate called after throwing an instance of 'std::bad_alloc'
node2: what(): std::bad_alloc
node2: [2023-04-26 06:34:37,245] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 1395
node2: [2023-04-26 06:34:37,247] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 1396
node2: [2023-04-26 06:34:37,247] [ERROR] [launch.py:434:sigkill_handler] ['/opt/conda/bin/python', '-u', 'main.py', '--local_rank=1', '--data_path', 'Dahoas/rm-static', '--data_split', '2,4,4', '--model_name_or_path', '/workspace/models/opt-1.3b', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--max_seq_len', '512', '--learning_rate', '9.65e-6', '--weight_decay', '0.1', '--num_train_epochs', '2', '--gradient_accumulation_steps', '1', '--lr_scheduler_type', 'cosine', '--num_warmup_steps', '0', '--seed', '1234', '--zero_stage', '3', '--deepspeed', '--output_dir', './output'] exits with return code = -6
pdsh@4e68f64d7185: node2: ssh exited with exit code 250

Launcher context
the ShmSize (shm) size is 10G
deepspeed --hostfile=hostfile
--master_port xxx --master_addr xxx
main.py ....

Both of my nodes can communicate with each other, and they are running inside docker containers. Have you found a solution to this issue yet?

[xiyue961]

Any updates? I encounted the same problem when finetuning whisper using deepspeed and multiple nodes.