Training loop temporarily hangs after every 4 steps

[VitorGuizilini-TRI]

I am porting some of my code to pytorch lightning, and everything seems to work fine. However, for some reason after every 4 training steps I see some temporary hanging (~1 second), which is severely slowing down my overall training time. Am I missing some obvious configuration? This is my Trainer configuration:

    trainer = pl.Trainer(
        gpus=8
        num_nodes=1,
        distributed_backend='ddp',
        checkpoint_callback=False,
        max_epochs=50,
        max_steps=None,
        progress_bar_refresh_rate=1,
        check_val_every_n_epoch=1,
        val_check_interval=1.0,
        gradient_clip_val=0.0,
        log_save_interval=0,
        num_sanity_val_steps=0,
        amp_level='O0',
    )

[github-actions]

Hi! thanks for your contribution!, great first issue!

[williamFalcon]

@PyTorchLightning/core-contributors

[ethanwharris]

Thanks for the issue! Would it be possible to post the code that reproduces this error? I've only seen this sort of behaviour before when the number of data loading workers is low - are you working with large data here (e.g. big images)?

[VitorGuizilini]

I increased the number of workers and it works perfectly now, thank you very much! You can close this issue.

[williamFalcon]

should we throw a warning when users use few workers?

[VitorGuizilini]

If possible, sure! Seems like an obvious solution now, but it could save a couple of hours for other people. :)