DDP doesn't work properly with CUDA_VISIBLE_DEVICE

[rebryk]

🐛 Bug

DDP trainer doesn't work properly when CUDA_VISIBLE_DEVICE is set.

To Reproduce

Steps to reproduce the behavior:

1. Set CUDA_VISIBLE_DEVICE=1,2
2. Run DDP trainer with 2 GPUs
3. The main process will use available_gpus[self.trainer.local_rank] that is equal to 1
4. The second process will use GPU process_idx that is again equal to 1
5. Thus both processes will use the same single GPU, instead of both

Expected behavior

Training should use both GPUs.

Suggestion

Looks like the following if-statement causes the problem:
https://github.com/PyTorchLightning/pytorch-lightning/blob/5b4db52851000d5e4eca8c680d851bcdaafc3a80/pytorch_lightning/accelerators/ddp_backend.py#L204

Why do we use int(available_gpus[self.trainer.local_rank]) instead of simple process_idx?
As far as I understand, the master process should always use GPU 0, which is equal to the first GPU in the CUDA_VISIBLE_DEVICE list. Please, correct me if I am wrong.

[github-actions]

Hi! thanks for your contribution!, great first issue!