C96C48_hybatmaerosnowDA fails with updated config.aeroanlgenb

[RussTreadon-NOAA]

What is wrong?

Job gdas_aeroanlgenb fails when running g-w CI case C96C48_hybatmaerosnowDA.

What should have happened?

All jobs in g-w CI case C96C48_hybatmaerosnowDA should successfully run to completion.

What machines are impacted?

All or N/A

What global-workflow hash are you using?

ba09498

Steps to reproduce

1. clone and build g-w develop at ba09498. Include gdas in the build.
2. set up and run g-w CI case C96C48_hybatmaerosnowDA
3. Job gdas_aeroanlgenb for 20211220 12Z will fail

Additional information

Via a series of tests documented in GDASApp issue #1510 it was found that changes to parm/config/gfs/config.aeroanlgenb made in g-w PR #3309 do not work on C96C48_hybatmaerosnowDA.

It is not clear, at present, which of the modified parameter settings is responsible for the failure. It's also not clear if the modified parameters are resolution specific.

Do you have a proposed solution?

No response

[CoryMartin-NOAA]

It's seemingly related to this variable:
aero_diagb_smooth_horiz_iter

Not sure 1) why or 2) what an acceptable value besides 0 is for this low resolution

[RussTreadon-NOAA]

I found the same sensitivity. This surprised me. Are there any JEDI issues which might help us figure out what's going on?

While we dig for the reason, should we open a hotfix PR to close g-w issue #3358? An easy, but ugly, hotfix is to toggle the value of aero_diagb_smooth_horiz_iter based on the CASE

[CoryMartin-NOAA]

working on that now @RussTreadon-NOAA

[CoryMartin-NOAA]

https://github.com/CoryMartin-NOAA/global-workflow/tree/bugfix/aerolowres

[CoryMartin-NOAA]

did a very crude test and it works, but we should have a 2nd verified test before I open the PR

[RussTreadon-NOAA]

I cloned https://github.com/CoryMartin-NOAA/global-workflow.git on Hera. I checked out bugfix/aerolowres, built it, and triggered g-w CI case C96C48_hybatmaerosnowDA. Results will be posted to this issue once C96C48_hybatmaerosnowDA finishes.

[RussTreadon-NOAA]

C96C48_hybatmaerosnowDA CI is still running on Hera but the previously failed gdas_aeroanlgenb job passed in this rerun

Hera(hfe04):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_hybatmaerosnowDA_aerolowres$ rocotostat -d C96C48_hybatmaerosnowDA_aerolowres.db -w C96C48_hybatmaerosnowDA_aerolowres.xml -c 202112201200 -t gdas_aeroanlgenb
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202112201200        gdas_aeroanlgenb                     7178088           SUCCEEDED                   0         1         133.0

[CoryMartin-NOAA]

Thanks @RussTreadon-NOAA I think @ClaraDraper-NOAA may wrap this into her open PR to minimize further disruptions

[ClaraDraper-NOAA]

@RussTreadon-NOAA I've run the test on hera only, and it went through OK. Code is here:

/scratch2/BMC/gsienkf/Clara.Draper/gerrit-hera/PRtesting/global_workflow_dev2

I pushed the change into PR [3295] (#3295), which is ready to merge ( I hope!)

[RussTreadon-NOAA]

@ClaraDraper-NOAA. The change @CoryMartin-NOAA made to parm/config/gfs/config.aeroanlgenb at 3eb4954 differs from what you committed at c83379d. We need @CoryMartin-NOAA 's config.aeroanlgenb committed to your branch, ClaraDraper-NOAA:feature/soilanal_det_clean

[RussTreadon-NOAA]

Hera g-w CI

All jobs in the run of C96C48_hybatmaerosnowDA using the modified config.aeroanlgenb successfully ran to completion.

Hera(hfe01):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_hybatmaerosnowDA_aerolowres$ rocotostat -d C96C48_hybatmaerosnowDA_aerolowres.db -w C96C48_hybatmaerosnowDA_aerolowres.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Feb 24 2025 19:37:22    Feb 24 2025 20:40:20
202112201800        Done    Feb 24 2025 19:37:22    Feb 25 2025 00:55:19
202112210000        Done    Feb 24 2025 19:37:22    Feb 25 2025 01:35:21

Job gdas_aeroanlgenb is run in three cycles of C96C48_hybatmaerosnowDA

202112201200        gdas_aeroanlgenb                     7178088           SUCCEEDED                   0         1         133.0
202112201800        gdas_aeroanlgenb                     7181924           SUCCEEDED                   0         1         135.0
202112210000        gdas_aeroanlgenb                     7188495           SUCCEEDED                   0         1         128.0

The modified config.aeroanlgenb was added to g-w PR #3295 at f1e6776