[Bug] Gradients not synchronized

[mephisto28]

sd-scripts/fine_tune.py

Line 247 in 2a23713

text_encoder, unet = train_util.transform_if_model_is_DDP(text_encoder, unet)

I have not idea what this line is used for, but this unwrap DDP module so that the training process become unsynchronized, i.e, no gradient communication in multi-gpu training, each node trains independently with part of the data.

I ensure this by adding sleep in one of the worker and finding no hang in the main training process. And by deleting this line the job got properly synchronized.

[mephisto28]

With above mentioned line not deleted:

With above mentioned line deleted:

[kohya-ss]

Thank you for opening the issue.

I have not directly developed the training code for multiple GPUs, and have received PRs.

Would these PRs be a reference? #165 and #448