Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update JEDI hashes (20250225) #1475

Merged
merged 28 commits into from
Feb 25, 2025
Merged

Update JEDI hashes (20250225) #1475

merged 28 commits into from
Feb 25, 2025
+15 −25

Conversation

RussTreadon-NOAA
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Feb 2, 2025 •
edited
Loading

Description

Update select JEDI hashes on 20250225

Companion PRs

none

Issues

Resolves #1474

Automated CI tests to run in Global Workflow

  • atm_jjob
  • C96C48_ufs_hybatmDA
  • C96C48_hybatmaerosnowDA
  • C48mx500_3DVarAOWCDA
  • C48mx500_hybAOWCDA
  • C96C48_hybatmDA

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Feb 2, 2025
@RussTreadon-NOAA
Copy link
Contributor Author

This PR will be marked Ready for review once all GDASApp ctests have been run and pass on Hera, Hercules, and Orion and g-w CI passes on WCOSS2.

@RussTreadon-NOAA RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion hercules-GW-RT Queue for automated testing with global-workflow on Hercules labels Feb 2, 2025
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules orion-GW-RT-Running Automated testing with global-workflow running on Orion hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules orion-GW-RT Queue for automated testing with global-workflow on Orion hera-GW-RT Queue for automated testing with global-workflow on Hera labels Feb 2, 2025
@emcbot
Copy link

emcbot commented Feb 2, 2025

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Sun Feb  2 14:48:35 CST 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sun Feb  2 15:24:43 CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sun Feb  2 16:24:09 CST 2025
Tests: 100% tests passed, 0 tests failed out of 135

@emcbot emcbot added hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Feb 2, 2025
@emcbot
Copy link

emcbot commented Feb 2, 2025

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Sun Feb  2 20:56:24 UTC 2025 on hfe09
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sun Feb  2 21:42:53 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sun Feb  2 22:43:09 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 135

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Feb 2, 2025
@emcbot
Copy link

emcbot commented Feb 2, 2025

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Sun Feb  2 02:51:15 PM CST 2025 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sun Feb  2 03:55:49 PM CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sun Feb  2 05:38:19 PM CST 2025
Tests: 100% tests passed, 0 tests failed out of 135

@emcbot emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Feb 2, 2025
@RussTreadon-NOAA
Copy link
Contributor Author

WCOSS2 g-w CI

Clone g-w develop at 380946c on Cactus. Update sorc/gdas.cd to feature/stable-nightly at e6eafb0`.

All g-w components successfully build except GDASApp. The GDASApp build fails with

[100%] Linking CXX executable ../../bin/gdas_soca_error_covariance_toolbox.x
cd /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build/gdas/mains && /apps/ops/test/spack-stack-1.6.0-nco/envs/nco-intel-19.1.3.304/install/intel/19.1.3.304/cmake-3.23.1-chpcsen/bin/cmake -E remove /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build/bin/gdas_soca_error_covariance_toolbox.x
cd /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build/gdas/mains && /apps/ops/test/\
spack-stack-1.6.0-nco/envs/nco-intel-19.1.3.304/install/intel/19.1.3.304/cmake-3.23.1-chpcsen/bin/cmake -E cmake_link_script CMakeFiles/gdas_soca_error_covariance_toolbox.x.dir/link.txt --verbose=NO
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: ../../lib/libsaber.so: undefined reference to `atlas::grid::detail::partitioner::TransPartitioner::TransPartitioner()'
make[2]: *** [gdas/mains/CMakeFiles/gdas_soca_error_covariance_toolbox.x.dir/build.make:174: bin/gdas_soca_error_covariance_toolbox.x] Error 1
make[2]: Leaving directory '/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build'
make[1]: *** [CMakeFiles/Makefile2:28951: gdas/mains/CMakeFiles/gdas_soca_error_covariance_toolbox.x.dir/all] Error 2
make[1]: Leaving directory '/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build'
make: *** [Makefile:166: all] Error 2

Notice that Hera, Hercules, and Orion use more recent versions of atlas

./hera.intel.lua:load("atlas/0.35.1")
./orion.intel.lua:load("atlas/0.35.1")
./wcoss2.intel.lua:load("atlas/0.35.0")
./hercules.gnu.lua:load("atlas/0.36.0")
./hercules.intel.lua:load("atlas/0.36.0")

Might a more recent version of atlas resolve the
undefined reference to atlas::grid::detail::partitioner::TransPartitioner::TransPartitioner()'
message in libsaber.so?

Manually load GDASApp wcoss2.intel modulefile on Cactus. module spider atlas returns

---------------------------------------------------------------------------------------------------------------------------------
  atlas:
---------------------------------------------------------------------------------------------------------------------------------
     Versions:
        atlas/0.33.0
        atlas/0.35.0

Unfortunately, atlas/0.35.0 is the most recent version of atlas available in the current installation of spack-stack/1.6.0 on WCOSS2.

Note that saber PR #1001 contains references to TransPartitioner.h in WriteFields.cc.

As a test back up working copy of sorc/saber to hash d51284c6. This is the commit prior to saber PR #1001. feature/stable-nightly successfully builds on Cactus using this saber hash.

We need to reach out to the library time to request installation of a newer version of atlas on WCOSS2. This PR can not move forward until we can build and run feature/stable-nightly on WCOSS2.

Attention: @DavidNew-NOAA , @danholdaway , @CoryMartin-NOAA , @guillaumevernieres

@RussTreadon-NOAA
Copy link
Contributor Author

WCOSS2 g-w CI
As a test revert sorc/saber back to d51284c6, build GDASApp, and run g-w CI on Cactus. All configurations successfully run to completion.

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 03 2025 10:05:18    Feb 03 2025 11:15:25
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Feb 03 2025 10:05:22    Feb 03 2025 10:20:23
202103250000      Active    Feb 03 2025 10:05:22             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Feb 03 2025 10:05:24    Feb 03 2025 10:20:30
202103250000        Done    Feb 03 2025 10:05:24    Feb 03 2025 11:25:21
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 03 2025 10:05:39    Feb 03 2025 12:01:05
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 03 2025 10:05:26    Feb 03 2025 11:30:46
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Feb 03 2025 10:05:29    Feb 03 2025 10:20:40
202112210000        Done    Feb 03 2025 10:05:29    Feb 03 2025 12:30:37
202112210600        Done    Feb 03 2025 10:05:29    Feb 03 2025 12:15:35
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Feb 03 2025 10:05:31    Feb 03 2025 10:25:41
202112201800        Done    Feb 03 2025 10:05:31    Feb 03 2025 12:30:41
202112210000        Done    Feb 03 2025 10:05:31    Feb 03 2025 12:20:45
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Feb 03 2025 10:05:33    Feb 03 2025 10:20:49
202112210000        Done    Feb 03 2025 10:05:33    Feb 03 2025 12:05:39
202112210600        Done    Feb 03 2025 10:05:33    Feb 03 2025 12:10:44
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Feb 03 2025 10:05:36    Feb 03 2025 10:20:53
202402240000        Done    Feb 03 2025 10:05:36    Feb 03 2025 12:53:23
202402240600        Done    Feb 03 2025 10:05:36    Feb 03 2025 12:52:37
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_S2SWA_gefs_replay_ics_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202011010000        Done    Feb 03 2025 10:05:43    Feb 03 2025 10:56:14

@RussTreadon-NOAA RussTreadon-NOAA added the DO NOT MERGE PR is not ready to be merged yet label Feb 3, 2025
@RussTreadon-NOAA
Copy link
Contributor Author

While g-w based GDASAPpp ctests pass on Hera, Hercules, and Orion, we can not build feature/stable-nightly at e6eafb0 on WCOSS2. Starting with saber @ b85ece5 we need at least atlas/0.35.1. Currently the WCOSS2 spack-stack installation only has atlas versions up to 0.35.0.

Mark this PR DO NO MERGE until WCOSS2 spack stack is updated.

@RussTreadon-NOAA
Copy link
Contributor Author

Reapply label orion-GT-RW. This time feature/stable-nightly is pulled into $HOMEgfs. Not sure why this didn't happen on the first try.

@RussTreadon-NOAA
Copy link
Contributor Author

Hercules CI is progressing very slowly. As of Tue Feb 25 14:05:46 UTC 2025 nineteen CI jobs are pending in the Hercules batch queue. Hercules CI uses the da-cpu accounting code. Queue wait times are expected to remain high given that Hercules da-cpu has used 260.7% of its monthly quota. As a result the Hercules da-cpu fairshare is very low

=================================================================================================================
               Cluster Queried:                         hercules
                   Report Type:        Project Report for:da-cpu
                    Report Run: Tue 25 Feb 2025 11:55:06 AM  UTC
       Report Period Beginning: Sat 01 Feb 2025 12:00:00 AM  UTC
          Report Period Ending: Sat 01 Mar 2025 12:00:00 AM  UTC
  Percentage of Period Elapsed:                            87.5%
Percentage of Period Remaining:                            12.5%
=================================================================================================================
                      Machines:         hercules
   Initial Allocation in Hours:        1,019,178
    Net Allocation Adjustments:          -98,631
                                ----------------
           Adjusted Allocation:          920,547

               Core Hours Used:        2,400,022
      Windfall Core Hours Used:                0
                                ----------------
         Total Core Hours Used:        2,400,022

     Project Normalized Shares:         0.036702
            Project Fair Share:         0.097186
                  Project Rank:          352/363

  Percentage of Period Elapsed:            87.5%
Percentage of Period Remaining:            12.5%
 Percentage of Allocation Used:           260.7%

@emcbot
Copy link

emcbot commented Feb 25, 2025

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Tue Feb 25 11:59:03 UTC 2025 on hfe06
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Feb 25 12:43:01 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Tue Feb 25 14:32:40 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 137

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Feb 25, 2025
@emcbot
Copy link

emcbot commented Feb 25, 2025

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Tue Feb 25 07:10:02 AM CST 2025 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Feb 25 08:20:17 AM CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Tue Feb 25 09:03:24 AM CST 2025
Tests: 
Tests: see output at /work2/noaa/da/role-da/CI/orion/GDASApp/workflow/PR/1475/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Feb 25, 2025
@RussTreadon-NOAA
Copy link
Contributor Author

Failed to copy g-w PR #3295 config.aeroanlgenb into Orion $HOMEgfs when I reapplied the orion-GW-RT label. Therefore, the following jobs failed (as exepected)

 83/137 Test #2041: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlgenb_202112201200 .......***Failed  297.15 sec
 96/137 Test #2048: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlinit_202112201800 .......***Failed   96.04 sec
 99/137 Test #2049: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlvar_202112201800 ........***Failed  131.65 sec
101/137 Test #2050: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlfinal_202112201800 ......***Failed   63.43 sec

Remove orion-GW-RT-Failed label. Reapply orion-GW-RT label.

@RussTreadon-NOAA RussTreadon-NOAA added orion-GW-RT Queue for automated testing with global-workflow on Orion and removed orion-GW-RT-Failed Automated testing with global-workflow failed on Orion labels Feb 25, 2025
@emcbot emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Feb 25, 2025
@RussTreadon-NOAA
Copy link
Contributor Author

WCOSS2 CI
g-w CI results on Cactus are as follows

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 25 2025 13:10:26    Feb 25 2025 14:20:33
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Feb 25 2025 13:10:28    Feb 25 2025 13:30:37
202103250000        Done    Feb 25 2025 13:10:28    Feb 25 2025 15:07:54
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Feb 25 2025 13:10:30    Feb 25 2025 13:30:40
202103250000        Done    Feb 25 2025 13:10:30    Feb 25 2025 14:30:47
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 25 2025 13:10:31    Feb 25 2025 15:05:32
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 25 2025 13:10:34    Feb 25 2025 14:29:04
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Feb 25 2025 13:10:35    Feb 25 2025 13:30:52
202112210000        Done    Feb 25 2025 13:10:35    Feb 25 2025 15:35:30
202112210600        Done    Feb 25 2025 13:10:35    Feb 25 2025 15:15:45
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Feb 25 2025 13:10:37    Feb 25 2025 13:30:55
202112201800        Done    Feb 25 2025 13:10:37    Feb 25 2025 15:29:45
202112210000      Active    Feb 25 2025 13:10:37             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Feb 25 2025 13:10:39    Feb 25 2025 13:30:58
202112210000        Done    Feb 25 2025 13:10:39    Feb 25 2025 15:10:44
202112210600        Done    Feb 25 2025 13:10:39    Feb 25 2025 15:10:44
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Feb 25 2025 13:10:41    Feb 25 2025 13:31:02
202402240000        Done    Feb 25 2025 13:10:41    Feb 25 2025 16:10:41
202402240600        Done    Feb 25 2025 13:10:41    Feb 25 2025 15:59:28
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96mx100_S2S_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Feb 25 2025 13:10:43    Feb 25 2025 14:01:18

All g-w CI cases ran to completion except C96C48_hybatmaerosnowDA. The following jobs died

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_stable-nightly
202112210000            gdas_snowanl                   181611557                DEAD                   1         2          58.0
202112210000       enkfgdas_esnowanl                   181611559                DEAD                   1         2          79.0

Both jobs died with the error

Traceback (most recent call last):
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/ush/imsfv3_scf2ioda.py", line 16, in <module>
    import pyiodaconv.ioda_conv_engines as iconv
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/sorc/iodaconv/src/pyiodaconv/ioda_conv_engines.py", line 3, in <module>
    from pyioda import ioda_obs_space as ioda_os
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build/lib/python3.10/pyioda/ioda_obs_space/__init__.py", line 4, in <module>
    from pyioda import ioda
  File "/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build/lib/python3.10/pyioda/ioda/__init__.py", line 8, in <module>
    from ._ioda_python import *
ImportError: /opt/cray/pe/lib64/libmpifort_intel.so.12: undefined symbol: MPIX_Enqueue_send

This undefined error message occurred in the past. The following patch was added to ush/load_ufsda_modules.sh to get past this error

@@ -41,6 +41,7 @@ case "${MACHINE_ID}" in
       # TODO: Add path to GDASApp libraries and cray-mpich as temporary patches
       # TODO: Remove LD_LIBRARY_PATH lines as soon as permanent solutions are available
       export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HOMEgfs}/sorc/gdas.cd/build/lib"
+      export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/cray/pe/mpich/8.1.19/ofi/intel/19.0/lib"
     fi
     module load "${MODS}/${MACHINE_ID}"
     ncdump=$( command -v ncdump )

Add this patch to working copy of $HOMEgfs/ush/load_ufsda_modules.sh. Rewind and reboot 20211221 00Z gdas_snowanl and enkfgdas_esnowanl. Both jobs successfully ran to completion.

russ.treadon@clogin04:/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_stable-nightly> rocotostat -d C96C48_hybatmaerosnowDA_stable-nightly.db -w C96C48_hybatmaerosnowDA_stable-nightly.xml -c 202112210000 | grep snowanl
202112210000            gdas_snowanl                   181622125           SUCCEEDED                   0         1         182.0
202112210000       enkfgdas_esnowanl                   181623085           SUCCEEDED                   0         1         167.0

and g-w CI case C96C48_hybatmaerosnowDA completed all jobs

russ.treadon@clogin04:/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_stable-nightly> rocotostat -d C96C48_hybatmaerosnowDA_stable-nightly.db -w C96C48_hybatmaerosnowDA_stable-nightly.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Feb 25 2025 13:10:37    Feb 25 2025 13:30:55
202112201800        Done    Feb 25 2025 13:10:37    Feb 25 2025 15:29:45
202112210000        Done    Feb 25 2025 13:10:37    Feb 25 2025 17:17:25

The load_ufsda_modules.sh patch was no longer needed after RFC 13644 was implemented 5-6 February 2025. Seems like something changed on Cactus which requires the patch to be added back ... at least for the snowanl jobs.

Attention: @CatherineThomas-NOAA , @ClaraDraper-NOAA , @CoryMartin-NOAA

@emcbot
Copy link

emcbot commented Feb 25, 2025

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Tue Feb 25 09:28:15 AM CST 2025 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Feb 25 10:36:55 AM CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Tue Feb 25 12:27:38 PM CST 2025
Tests: 100% tests passed, 0 tests failed out of 137

@emcbot emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Feb 25, 2025
@emcbot
Copy link

emcbot commented Feb 25, 2025

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Tue Feb 25 05:52:54 CST 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Feb 25 06:29:04 CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Tue Feb 25 13:49:41 CST 2025
Tests: 100% tests passed, 0 tests failed out of 137

@emcbot emcbot added hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Feb 25, 2025
@RussTreadon-NOAA RussTreadon-NOAA changed the title Update JEDI hashes (20250203) Update JEDI hashes (20250225) Feb 25, 2025
@RussTreadon-NOAA RussTreadon-NOAA marked this pull request as ready for review February 25, 2025 19:59
@RussTreadon-NOAA
Copy link
Contributor Author

GDASApp CI passed on Hera, Hercules, and Orion. g-w CI passed on WCOSS2 (Cactus) with caveats noted above. This PR is ready for review.

@RussTreadon-NOAA
Copy link
Contributor Author

Thank you @CoryMartin-NOAA and @DavidNew-NOAA for the quick approvals. Now merging.

@RussTreadon-NOAA RussTreadon-NOAA merged commit 583e630 into develop Feb 25, 2025
33 checks passed
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/stable-nightly branch February 25, 2025 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CoryMartin-NOAA CoryMartin-NOAA CoryMartin-NOAA approved these changes

@DavidNew-NOAA DavidNew-NOAA DavidNew-NOAA approved these changes

@danholdaway danholdaway danholdaway left review comments

@guillaumevernieres guillaumevernieres Awaiting requested review from guillaumevernieres

Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels
hera-GW-RT-Passed Automated testing with global-workflow successful on Hera hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules orion-GW-RT-Passed Automated testing with global-workflow successful on Orion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

gdas_aeroanlgenb failure in 20250222 stable-nightly testing Update JEDI hashes (20250225)
6 participants
@RussTreadon-NOAA @emcbot @CoryMartin-NOAA @shlyaeva @danholdaway @DavidNew-NOAA