Remove RUNDIRS before running CI cases to cover re-run events

[TerrenceMcGuinness-NOAA]

Description

This Pr removes pslot dir (with hash) from local archive folder on CI case completion and also adds cleanup_experiment function in the BASH utilities to clean up after a CI case runs. This change consolidates cleanup actions and adds functionality to clean the local archive folders.

Key changes:

• Fixes bug for cleaning up the RUNDIRS directory before new experiments are run in the case the pipeline is being ran for a second time
• Updated $HOMEgfs/ci/scripts/check_ci.sh to use the new cleanup_experiment function under the BASH CI system.
• Added cleanup_experiment function in $HOMEgfs/ci/scripts/utils/ci_utils.sh to handle various cleanups including:
  • COMROOT/EXPDIR per case
  • ARCDIR and ATADIR directories for archive
  • STMP/RUNDIRS/${PSLOT}
• Added get config var from EXPDIR utility get_config_var.py for getting config values in BASH

Resolves #2954, #3066

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
  • EMC verif-global
  • GDAS
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

This PR still needs to be tested with the CI system which would effectively be a self-test.
# Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• I have made corresponding changes to the system documentation if necessary

[emcbot]

Checkout Failed on Hera in Build# 2: Error cloning remote repo 'origin'

[TerrenceMcGuinness-NOAA]

Checkout failed for gfs but not for gefs on a basic git command on Hera. Simply restarting CI in attempts to repeat the error.
Error was not repeatable for the gfs checkout and proceeded normally. scm also did the checkout in normal git fashion with updates only on previously gefs side as well.

[emcbot]

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3076

Experiment C48_ATM_6d349326 Completed 2 Cycles: *SUCCESS* at Thu Nov  7 13:41:03 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_6d349326 Completed 1 Cycles: *SUCCESS* at Thu Nov  7 14:05:23 CST 2024
Experiment C48_S2SW_6d349326 Completed 2 Cycles: *SUCCESS* at Thu Nov  7 15:06:18 CST 2024
Experiment C96_atm3DVar_6d349326 Completed 3 Cycles: *SUCCESS* at Thu Nov  7 15:18:15 CST 2024
Experiment C48_S2SWA_gefs_6d349326 Completed 1 Cycles: *SUCCESS* at Thu Nov  7 15:31:08 CST 2024
Experiment C96C48_hybatmDA_6d349326 Completed 3 Cycles: *SUCCESS* at Thu Nov  7 15:42:48 CST 2024

[emcbot]

CI Passed on Hercules in Build# 17
Built and ran in directory /work2/noaa/global/CI/HERCULES/3076

Experiment C48_ATM_1e17407f Completed 2 Cycles: *SUCCESS* at Fri Nov  8 22:31:17 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_1e17407f Completed 1 Cycles: *SUCCESS* at Fri Nov  8 22:43:24 CST 2024
Experiment C48_S2SWA_gefs_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 00:14:57 CST 2024
Experiment C96_atm3DVar_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 00:26:20 CST 2024
Experiment C48_S2SW_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 00:38:24 CST 2024
Experiment C96C48_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 00:50:51 CST 2024

[emcbot]

CI Passed on Hera in Build# 18
Built and ran in directory /scratch1/NCEPDEV/global/CI/3076

Experiment C48_ATM_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 07:18:41 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 07:49:07 UTC 2024
Experiment C48mx500_3DVarAOWCDA_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 08:19:42 UTC 2024
Experiment C48_S2SWA_gefs_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 09:40:22 UTC 2024
Experiment C48_S2SW_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 09:51:05 UTC 2024
Experiment C96C48_hybatmaerosnowDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 10:45:46 UTC 2024
Experiment C96_atm3DVar_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 10:51:50 UTC 2024
Experiment C96C48_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 11:04:37 UTC 2024
Experiment C96C48_ufs_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 12:11:02 UTC 2024

[emcbot]

Checkout Failed on Orion in Build# 19: Error fetching remote repo 'origin'

[emcbot]

Checkout Failed on Orion in Build# 19: Error fetching remote repo 'origin'

[TerrenceMcGuinness-NOAA]

Massive failure of scm checkout of global-workflow on Orion.
Looking into a lead with this part of the error log.

fatal: write error: Disk quota exceeded
fatal: fetch-pack: invalid index-pack output

[WalterKolczynski-NOAA]

PR description needs to be updated before merging.