Skip to content

Latest commit

 

History

History
19 lines (12 loc) · 608 Bytes

PipeDebug.md

File metadata and controls

19 lines (12 loc) · 608 Bytes

Debugging

  • debugging mpi python applications with vscode

    • debug work only when dataloading is on main thread. (num_data_workers=0).
    • run same thing, with --debug flag, then wait for attachment:
    mpirun -np 2 python main.py --debug <LIST OF RANKS>

  • If you debug cuda, you may want to fix the trace by:

    CUDA_LAUNCH_BLOCKING=1 mpirun -np 2 python main.py --debug <LIST OF RANKS>

  • Before you debug, you may want to check run the error is cuda specific and not cpu