FV3PETS Calculation revisited, or srun argument "--ntasks" now includes num_threads?

[mdtoyNOAA]

Description
This issue is a question about the meaning of "number of tasks" in relation to the Slurm "srun" argument "--ntasks". I could be wrong, but in earlier versions of the global workflow (prior to ~May 2023) "ntasks" equalled the number of MPI tasks calculated from "6 times layout_x times layout_y + write_groups times write_tasks_per_group". In the latest versions of the global workflow, "ntasks" is now the number of MPI tasks multiplied by the number of threads. Am I interpreting this correctly? And if so, why the change? Does each MPI task now get it's own separate "sub-task" for each thread?

Also, does threading apply to the I/O tasks?

BTW, I believe this issue is related to the "FV3PETS Calculation" issue #1583. And I imagine the change has to do with the transition from OMP to ESMF threading.

[aerorahul]

@mdtoyNOAA
The use of ESMF-based threading changed the resource specification for the ufs-weather-model.
I am tagging the relevant experts who can answer this much better than I can.
@junwang-noaa @DeniseWorthen @DusanJovic-NOAA @theurich

[DusanJovic-NOAA]

Yes, total number of tasks (for atm component) is equal to ((layout_x * layout_y * ntiles) + (write_groups * write_tasks_per_group) ) * number_of_threads, if ESMF managed threading is used. Threading also applies to the write components (I/O tasks). This was introduced to allow different components to run with different number of threads, or without threading (i.e. one thread).

[mdtoyNOAA]

Yes, total number of tasks (for atm component) is equal to ((layout_x * layout_y * ntiles) + (write_groups * write_tasks_per_group) ) * number_of_threads, if ESMF managed threading is used. Threading also applies to the write components (I/O tasks). This was introduced to allow different components to run with different number of threads, or without threading (i.e. one thread).

Thank you, @DusanJovic-NOAA

[yangfanglin]

@DusanJovic-NOAA In this case, isn't the ATM is still using the same threading as the Write Gird Component ? I thought one of the major issues was that the Write Grid Component cannot use different threading from the ATM forecast model.

[DusanJovic-NOAA]

Both fv3atm subcomponents (fcst and write grid component) use the same number of threads.