Debugging Tools

General steps to debug GEOSldas using applications such as Totalview or DDT :

1. Compile GEOSldas for debugging: parallel_build.csh -debug. This will build GEOSldas in ./build-Debug/ and install into ./install-Debug/.

2. Follow the README.md instructions to set up and run the experiment except for the final step (i.e., do not run sbatch lenkf.j). Instead, execute lenkf.j -debug, which will complete some preprocessing and then stop at the point where GEOSldas.x would normally be run.

3. Go to the experiment's ./scratch directory, source g5_modules, and launch the debugging tool.

4. Configure/verify the debugger options, including the MPI stack and number of tasks, which must match that in lenkf.j.

Note that the above instructions include setting up a new experiment. If you want to make changes to your debugging configuration in Steps 3 and 4, a new experiment setup may be required.

Using Totalview at NCCS:

This section provides more detailed but NCCS-specific instructions for Steps 3 and 4 using Totalview.

To use debugging tools at NCCS, you must be on an interactive compute node, which can be obtained with the following sample command:

xalloc --ntasks=28 --time=01:00:00 --account=[NCCS_COMPUTATIONAL_PROJECT_NUMBER]

On the interactive compute node, execute:

cd [SCRATCHDIR]
source [PATH]/g5_modules
module load tview
totalview [PATH]/GEOSldas.x &

where [SCRATCHDIR] is the experiment's scratch directory and [PATH] points to the ./install-Debug/bin/ directory of step 1 above.

This will open two Totalview windows. Click on the larger Totalview window (displaying source code) and press Ctrl-A to open the "Startup Parameters" window. (Alternatively, the "Startup Parameters" window can be selected from the "Process" tab.)

Use the tabs in the "Startup Parameters" window to edit/verify the Totalview configuration.
For "Parallel" debugging:

• The number of tasks must match that in lenkf.j.
• The selected MPI stack must be consistent with the compiler and MPI stack encoded in the ./install-Debug/bin/g5_modules script. As of October 2022, Intel MPI-Hydra is consistent with the Intel compiler selected in g5_modules.

Users should also refer to the following NCCS documentation:

• available debugging tools and configuration instructions
• "module" command

Note: Debuggers may run out of licenses if too many processors are used; 28 processors are recommended.