Stat Analysis Implementation for global-workflow #8

kevindougherty-noaa · 2024-11-20T19:57:50Z

Below are the appropriate changes to run a stat analysis job in the global-workflow.

Additional changes to GDASApp are also needed for jcb-gdas and jcb-algorithms for these scripts to run completely successfully.

kevindougherty-noaa · 2024-12-12T20:30:26Z

@CoryMartin-NOAA you will need to add Ed and Andrew to be able to review whenever you get a chance.

CoryMartin-NOAA · 2024-12-13T15:19:18Z

@ADCollard @EdwardSafford-NOAA feel free to take a look at this. I'll do the same and once we've done a review/test, we will open the PR with the authoritative global-workflow repo.

This reverts commit 899b20b.

kevindougherty-noaa · 2025-01-28T16:34:41Z

ush/python/pygfs/task/stat_analysis.py

+            # Get list of .nc4 files
+            # obs_space_paths = glob.glob(os.path.join(ob_dir_str, "*.{nc,nc4}")) # THIS SHOULD WORK BUT ISNT, glob patterns introduced in Python 3.9
+            nc_paths = glob.glob(os.path.join(ob_dir_str, "*.nc"))
+            nc4_paths = glob.glob(os.path.join(ob_dir_str, "*.nc4"))
+            obs_space_paths = nc_paths + nc4_paths
+
+            # Temporary. Create condition check here for available jcb algorithms?
+            if OB == 'snow':
+                obs_space_paths = glob.glob(os.path.join(ob_dir_str, "diag_ims_snow_*.nc"))
+
+            # This grabs the obspace string from the .nc4 files, however not all are perfect. Need solution.
+            self.task_config.OBSPACES_LIST = ['_'.join(os.path.basename(path).split('_')[1:3]) for path in obs_space_paths]


Cory and I were discussing this offline. Maybe would be best to have some sort of configuration file to avoid grabbing the wrong files. Also not all file names follow the same format mentioned in comment on line 133. This would remove the temporary fix on lines 130-131.

Yes this is too rigid and prone to failure, this and the above globs are risky. We should have a list of files that are expected based on some configuration file

kevindougherty-noaa · 2025-01-28T16:35:47Z

ush/python/pygfs/task/stat_analysis.py

+            # Gunzip .nc files
+            logger.info("Gunzip files from tar file")
+            gz_files = glob.glob(os.path.join(ob_dir_str, "*.gz"))
+            logger.info(f"Gunzip files: {gz_files}")
+
+            for diagfile in gz_files:
+                output_file = os.path.join(ob_dir_str, os.path.basename(diagfile)[:-3])
+                with gzip.open(diagfile, 'rb') as f_in:
+                    with open(output_file, 'wb') as f_out:
+                        f_out.write(f_in.read())


I believe I need to have a check after this to make sure there are files in the tar file. If not, move on to next ob. What are you thoughts?

Yes, a check if it is non-empty is necessary

CoryMartin-NOAA · 2025-01-28T18:11:21Z

env/HERA.env

@@ -105,6 +105,11 @@ elif [[ "${step}" = "atmanlfv3inc" ]]; then
    export NTHREADS_ATMANLFV3INC=${NTHREADSmax}
    export APRUN_ATMANLFV3INC="${APRUN_default} --cpus-per-task=${NTHREADS_ATMANLFV3INC}"

+elif [[ "${step}" = "anlstat" ]]; then


we will need similar entries for the other machines when this goes into a PR, not just HERA

@kevindougherty-noaa we need FOO.env files for each supported machine (GAEA, HERCULES, ORION, WCOSS2) not just HERA

CoryMartin-NOAA

First pass of comments, @kevindougherty-noaa let me know if you need any clarifications or want to discuss anything

CoryMartin-NOAA · 2025-01-28T18:20:16Z

jobs/JGLOBAL_ANALYSIS_STATS

+##############################################
+
+# Generate COM variables from templates
+YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_OBS COM_CHEM_ANALYSIS


Workflow will need these modified to be COMIN not COM, also I suggest a cleaner declaration so that we can add each 'component' on a new line.

jobs/JGLOBAL_ANALYSIS_STATS

CoryMartin-NOAA · 2025-01-28T18:22:33Z

jobs/JGLOBAL_ANALYSIS_STATS

+  cat "${pgmout}"
+fi
+
+exit 0


CoryMartin-NOAA · 2025-01-28T18:23:11Z

parm/config/gfs/config.anlstat

+export JCB_BASE_YAML="${PARMgfs}/gdas/stat/aero/jcb-base.yaml.j2"
+export JCB_ALGO_YAML="${PARMgfs}/gdas/jcb-algorithms/anlstat.yaml.j2"
+export JEDIEXE=${HOMEgfs}/sorc/gdas.cd/build/bin/ioda-stats.x
+export STAT_OUTDIR="${COMOUT}/${net}/${run.yyyymmdd}/${hh}/products/stats/"


this gets replaced by COMOUT_ATMOS_ANALYSIS_STATS etc. or something similar

CoryMartin-NOAA · 2025-01-28T18:23:18Z

parm/config/gfs/config.anlstat

+export JEDIEXE=${HOMEgfs}/sorc/gdas.cd/build/bin/ioda-stats.x
+export STAT_OUTDIR="${COMOUT}/${net}/${run.yyyymmdd}/${hh}/products/stats/"
+
+echo "END: config.anlstat"


CoryMartin-NOAA · 2025-01-28T18:27:35Z

ush/python/pygfs/task/stat_analysis.py

+                'npx_ges': _res + 1,
+                'npy_ges': _res + 1,
+                'npz_ges': self.task_config.LEVS - 1,
+                'npz': self.task_config.LEVS - 1,


do we care about the number of levels?

CoryMartin-NOAA · 2025-01-28T18:28:09Z

ush/python/pygfs/task/stat_analysis.py

+
+        # Loop through ob space list
+        for OB in self.task_config.STAT_OBS:
+            logger.info(f"Working on current observation: {OB}")


this is snow and aero right? then it's not an 'observation'

What would be a good replacement for this? Maybe anl to fit with your above suggestion?

yes I'd suggest something like f"Working on current component: {anl}"

CoryMartin-NOAA · 2025-01-28T18:28:39Z

ush/python/pygfs/task/stat_analysis.py

+            # Gunzip .nc files
+            logger.info("Gunzip files from tar file")
+            gz_files = glob.glob(os.path.join(ob_dir_str, "*.gz"))
+            logger.info(f"Gunzip files: {gz_files}")
+
+            for diagfile in gz_files:
+                output_file = os.path.join(ob_dir_str, os.path.basename(diagfile)[:-3])
+                with gzip.open(diagfile, 'rb') as f_in:
+                    with open(output_file, 'wb') as f_out:
+                        f_out.write(f_in.read())


Yes, a check if it is non-empty is necessary

CoryMartin-NOAA · 2025-01-28T18:29:17Z

ush/python/pygfs/task/stat_analysis.py

+            # Get list of .nc4 files
+            # obs_space_paths = glob.glob(os.path.join(ob_dir_str, "*.{nc,nc4}")) # THIS SHOULD WORK BUT ISNT, glob patterns introduced in Python 3.9
+            nc_paths = glob.glob(os.path.join(ob_dir_str, "*.nc"))
+            nc4_paths = glob.glob(os.path.join(ob_dir_str, "*.nc4"))
+            obs_space_paths = nc_paths + nc4_paths
+
+            # Temporary. Create condition check here for available jcb algorithms?
+            if OB == 'snow':
+                obs_space_paths = glob.glob(os.path.join(ob_dir_str, "diag_ims_snow_*.nc"))
+
+            # This grabs the obspace string from the .nc4 files, however not all are perfect. Need solution.
+            self.task_config.OBSPACES_LIST = ['_'.join(os.path.basename(path).split('_')[1:3]) for path in obs_space_paths]


Yes this is too rigid and prone to failure, this and the above globs are risky. We should have a list of files that are expected based on some configuration file

CoryMartin-NOAA · 2025-01-28T18:29:57Z

ush/python/pygfs/task/stat_analysis.py

+
+    @logit(logger)
+    def finalize(self, jedi_dict_key: str) -> None:
+        """Finalize a statistic analysis


need to reword these, as 'finalize a statistic analysis' doesn't make sense

CoryMartin-NOAA · 2025-01-28T18:31:13Z

tagging @EdwardSafford-NOAA for awareness

Co-authored-by: Cory Martin <[email protected]>

CoryMartin-NOAA · 2025-01-28T19:50:29Z

@RussTreadon-NOAA FYI

…-NOAA/global-workflow into feature/stat-analysis

kevindougherty-noaa · 2025-02-04T16:18:32Z

@CoryMartin-NOAA @EdwardSafford-NOAA I have addressed the following requests made for the accompanying scripts. However, I would like to open the discussion on how best to handle Cory's comment on how a config file should be created for the issue in the stat_analysis.py file. Where and how should this config file be stored? What is the best method to use this? Like Cory mentioned, globs can be risky and all the files do not follow the same naming structures. What are your thoughts?

CoryMartin-NOAA · 2025-02-04T19:17:25Z

@kevindougherty-noaa can you put a YAML in the config directory that is something like this:

atmos:
  obs spaces:
  - name:
     input file:
     config yaml:
aero:
  obs spaces:
  - name:
     input file:
     config yaml:

that then gets templated by Jinja? Would that be viable?

kevindougherty-noaa · 2025-02-14T15:21:22Z

jobs/JGLOBAL_ANALYSIS_STATS

+YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COMIN_OBS:COM_OBS_TMPL \
+                                           COMOUT_CONF:COM_CONF_TMPL \
+                                           COMIN_ATMOS_ANALYSIS:COM_ATMOS_ANALYSIS_TMPL \
+                                           COMIN_OCEAN_ANALYSIS:COM_OCEAN_ANALYSIS_TMPL \
+                                           COMIN_CHEM_ANALYSIS:COM_CHEM_ANALYSIS_TMPL \
+                                           COMIN_SNOW_ANALYSIS:COM_SNOW_ANALYSIS_TMPL


Need to double check to make sure these work.

kevindougherty-noaa added 7 commits November 20, 2024 19:28

Update and add scripts for analysis stats capabilities

c7fa90a

pynorms

b154521

pynorms

e3d015a

update missed files

f745d3d

update task names to include underscores

c04eb6a

solved issue of job not running, files in wrong directory

cccd670

update to include execute and finalize

cfb9f25

kevindougherty-noaa mentioned this pull request Dec 10, 2024

Jedi Statistic Analysis Job NOAA-EMC/global-workflow#3147

Open

5 tasks

updated scripts to that run successfully

899b20b

kevindougherty-noaa requested a review from CoryMartin-NOAA December 12, 2024 20:29

Merge branch 'develop' into feature/stat-analysis

164f7e4

This was referenced Dec 16, 2024

Stat Analysis Implementation #7

Closed

Stat Analysis Implementation #6

Closed

kevindougherty-noaa added 15 commits December 16, 2024 15:15

merge develop

3d09d46

Merge branch 'develop' into HEAD

452aff0

update files

840153a

fix merge conflicts

ae48429

Revert "updated scripts to that run successfully"

ba89a3f

This reverts commit 899b20b.

updated scripts to that run successfully

5b8e7b1

reverting commits that were causing issues

6bcd080

fix descrepancies with workflow/applications files

76e60b1

pycodestyle

332cede

pynorms

74c7888

add snow jcb-base yaml

6687c41

update aero jcb-base yaml to no longer include variables

cb4a703

small bug fixes

d0a7a3d

merge develop

e1209dd

merge develop

254ccc9

kevindougherty-noaa added 4 commits January 8, 2025 20:24

update workflow tasks correctly

fff3e76

update gfs tasks

32b8a7a

updates to python scripts to handle snow ob spaces

9fe7e20

fix pycodestyle issues

1454c0f

kevindougherty-noaa marked this pull request as ready for review January 28, 2025 16:29

kevindougherty-noaa commented Jan 28, 2025

View reviewed changes

CoryMartin-NOAA reviewed Jan 28, 2025

View reviewed changes

kevindougherty-noaa and others added 4 commits January 28, 2025 14:01

Update jobs/JGLOBAL_ANALYSIS_STATS

e05e241

Co-authored-by: Cory Martin <[email protected]>

Update scripts/exglobal_analysis_stats.py

6cfcc48

Co-authored-by: Cory Martin <[email protected]>

Update scripts/exglobal_analysis_stats.py

7d4b161

Co-authored-by: Cory Martin <[email protected]>

basic updated suggestions i.e. variable changes, EOL, etc.

388c58b

kevindougherty-noaa added 2 commits January 28, 2025 19:51

Merge branch 'feature/stat-analysis' of https://github.com/CoryMartin…

74974a9

…-NOAA/global-workflow into feature/stat-analysis

added checks to python scripts. fixed formatting of job

4055c99

kevindougherty-noaa added 2 commits February 14, 2025 15:16

update to utilize config file in driver python script

f8f6a7b

pycodestyle

b53a9d8

kevindougherty-noaa requested a review from CoryMartin-NOAA February 14, 2025 15:20

kevindougherty-noaa commented Feb 14, 2025

View reviewed changes

kevindougherty-noaa and others added 8 commits February 14, 2025 15:50

Merge branch 'develop' into feature/stat-analysis

69e06b8

remove STAT_OUTDIR from shell, need to still do it in py

f75140c

use different names for the output directory

0676314

a todo note

d8994cd

minor change and add outdir check

e1f04f3

Merge branch 'develop' into feature/stat-analysis

3082f41

Merge branch 'develop' into feature/stat-analysis

eb6cb96

minor updates with name changes and new aero tar file

54c6946

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stat Analysis Implementation for global-workflow #8

Stat Analysis Implementation for global-workflow #8

kevindougherty-noaa commented Nov 20, 2024

kevindougherty-noaa commented Dec 12, 2024

CoryMartin-NOAA commented Dec 13, 2024

kevindougherty-noaa Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

kevindougherty-noaa Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Feb 19, 2025

CoryMartin-NOAA left a comment

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

kevindougherty-noaa Jan 28, 2025 •

edited

Loading

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA Jan 28, 2025

CoryMartin-NOAA commented Jan 28, 2025

CoryMartin-NOAA commented Jan 28, 2025

kevindougherty-noaa commented Feb 4, 2025

CoryMartin-NOAA commented Feb 4, 2025

kevindougherty-noaa Feb 14, 2025

Stat Analysis Implementation for global-workflow #8

Are you sure you want to change the base?

Stat Analysis Implementation for global-workflow #8

Conversation

kevindougherty-noaa commented Nov 20, 2024

kevindougherty-noaa commented Dec 12, 2024

CoryMartin-NOAA commented Dec 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevindougherty-noaa Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CoryMartin-NOAA commented Jan 28, 2025

CoryMartin-NOAA commented Jan 28, 2025

kevindougherty-noaa commented Feb 4, 2025

CoryMartin-NOAA commented Feb 4, 2025

Choose a reason for hiding this comment

kevindougherty-noaa Jan 28, 2025 •

edited

Loading