One of the variables needed for gradient computation has been modified by an inplace operation

[jimmy-dq]

Hi,
Thanks for your great work. I ran rocl_train.py with your default parameters and find one error output when executing loss.backward(), the error is:
_RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.
I found the problem is in adv_loss. After we execute outputs = projector(model(inputs)), adv_loss cannot be backwarded due to the inplace operation. Appreciate for your help.

[immuntasir]

Hi, were you able to solve this?

[cfeng16]

Hi, I met the same problem, Have you solved it?

[ramithuh]

@immuntasir were you able to solve it ? 😃

[priyamdey]

It runs fine with multiple gpus. Error pops up for single gpu run. Comment out the lines 88 to 100 which wraps the model with SyncBN & DDP. Also the init (line 47 to 52, although I haven't tried with keeping this intact). Accordingly, update the run command by removing the distributed part. It should work fine then.