Enable cycling support for Gaea C6

[DavidHuber-NOAA]

Description

This adds support for the Gaea clusters by enabling cycled experiments on C6 and addressing a number of issues on both clusters.

Included in this PR is support for HPSS on C6, but it should NOT be utilized at this time. The ES cluster, where HPSS connections are made, has a significant issue with the F6 (C6's filesystem) mount that causes the system to run extremely slow when filesystem-intense operations are performed (such as htar). There is a plan to enable this feature more broadly in the near future.

• Fixed memory variable unsetting for Gaea C5/6 in config.resources.GAEAC{5,6}
• Refactoring the system-level parameter detection when determining task resources in the setup scripts to make it easier to define multiple partitions, queues, and clusters.
• Adding a DTN partition, queue, and cluster definition.
• Added/renamed missing/miss-named tasks to tasks.py and added a check that the input task is valid.

NOTE: Archiving from the DTNs for files located on the f6 filesystem is excruciatingly slow and can bog down both C5 and C6. Thus, it is recommended to not use HPSS at this time on Gaea/C6. Therefore, the option is disabled by default. According to system admins, there should be new DTNs installed soon that will help alleviate this issue.

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

• C48_ATM on C6
• C48_S2SW on C6
• Cycle testing on C6
• CI suite on Hera
• CI suite on WCOSS2

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[DavidHuber-NOAA]

Updated the title and description of this PR. Launching CI on Hercules.

[emcbot]

Build FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/global/CI/HERCULES/3323/global-workflow/sorc/logs/gefs_ww3_prepost.log
/work2/noaa/global/CI/HERCULES/3323/global-workflow/sorc/logs/gsi_monitor.log

Follow link here to view the contents of the above file(s): (link)

[KateFriedman-NOAA]

Thanks for putting all of these C6 updates together @DavidHuber-NOAA !

[aerorahul]

looks good @DavidHuber-NOAA and @KateFriedman-NOAA