Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove on disk archive files in CI pipeline after running a test #2954

Closed
TerrenceMcGuinness-NOAA opened this issue Sep 24, 2024 · 1 comment · Fixed by #3076
Closed

Remove on disk archive files in CI pipeline after running a test #2954

TerrenceMcGuinness-NOAA opened this issue Sep 24, 2024 · 1 comment · Fixed by #3076
Assignees
@TerrenceMcGuinness-NOAA
Labels
CI/CD Issue related to CI/CD feature New feature or request

Comments

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

What new functionality do you need?

As a CI user I need to have the on disk archive files removed after a CI test case passes.

What are the requirements for the new functionality?

Need to remove the directory $HOMEDIR/archive/$PSLOT after each CI case completes successfully

Acceptance Criteria

We see the folder is gone after the tests runs

Suggest a solution (optional)

No response
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added feature New feature or request triage Issues that are triage labels Sep 24, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI/CD Issue related to CI/CD and removed triage Issues that are triage labels Sep 24, 2024
Closed
12 tasks
@TerrenceMcGuinness-NOAA
Copy link
Collaborator Author

TerrenceMcGuinness-NOAA commented Nov 4, 2024 •
edited
Loading

A suggestion was made to simply scape the paths from the actual archive moves from the log file in COMROOT (example below)
This still needs to be implemented.

mterry (hercules-login-4) COMROOT $ echo $COMROOT
C96_S2SWA_gefs_replay_ics_32b8cd0c

mterry (hercules-login-4) COMROOT $ sed -n "s/file_utils  : Copied [[:graph:]]* to \([[:graph:]]*\)/\1/p" "${COMROOT}"/logs/*/*arch.log | awk '{print $NF}' | uniq
/work2/noaa/stmp/CI/HERCULES/2961/RUNTESTS/COMROOT/C96_S2SWA_gefs_replay_ics_32b8cd0c/gefsarch

mterry (hercules-login-4) COMROOT $ ls /work2/noaa/stmp/CI/HERCULES/2961/RUNTESTS/COMROOT/C96_S2SWA_gefs_replay_ics_32b8cd0c/gefsarch
gefs.t00z.mean.pres_.1p00.f006.grib2  gefs.t00z.mean.pres_.1p00.f012.grib2  gefs.t00z.mean.pres_.1p00.f018.grib2  gefs.t00z.mean.pres_.1p00.f024.grib2	gefs.t00z.mean.pres_.1p00.f030.grib2  gefs.t00z.mean.pres_.1p00.f036.grib2  gefs.t00z.mean.pres_.1p00.f042.grib2  gefs.t00z.mean.pres_.1p00.f048.grib2

Merged
21 tasks
WalterKolczynski-NOAA pushed a commit that referenced this issue Nov 13, 2024
@TerrenceMcGuinness-NOAA
Remove RUNDIRS before running CI cases to cover re-run events (#3076)
1872499 
This Pr removes **pslot** dir (with hash) from local archive folder on
CI case completion and also adds `cleanup_experiment` function in the
BASH utilities to clean up after a CI case runs. This change
consolidates cleanup actions and adds functionality to clean the local
archive folders.

Key changes:

- Fixes bug for cleaning up the `RUNDIRS` directory before new
experiments are run in the case the pipeline is being ran for a second
time
- Updated `$HOMEgfs/ci/scripts/check_ci.sh` to use the new
`cleanup_experiment` function under the BASH CI system.
- Added `cleanup_experiment `function in
`$HOMEgfs/ci/scripts/utils/ci_utils.sh` to handle various cleanups
including:
  - COMROOT/EXPDIR per case
  - ARCDIR and ATADIR directories for archive
  - STMP/RUNDIRS/${PSLOT}
- Added get config var from EXPDIR utility
[get_config_var.py](https://github.com/NOAA-EMC/global-workflow/pull/2961/files#diff-b780733ae1d45917730364e09a6c510e79d4cc8cffad6d9020c7961c53b987bc)
for getting config values in BASH

Resolves #2954
Resolves #3066
WalterKolczynski-NOAA pushed a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Nov 22, 2024
@TerrenceMcGuinness-NOAA @WalterKolczynski-NOAA
Remove RUNDIRS before running CI cases to cover re-run events (NOAA-E…
a87bcde 
…MC#3076)

This Pr removes **pslot** dir (with hash) from local archive folder on
CI case completion and also adds `cleanup_experiment` function in the
BASH utilities to clean up after a CI case runs. This change
consolidates cleanup actions and adds functionality to clean the local
archive folders.

Key changes:

- Fixes bug for cleaning up the `RUNDIRS` directory before new
experiments are run in the case the pipeline is being ran for a second
time
- Updated `$HOMEgfs/ci/scripts/check_ci.sh` to use the new
`cleanup_experiment` function under the BASH CI system.
- Added `cleanup_experiment `function in
`$HOMEgfs/ci/scripts/utils/ci_utils.sh` to handle various cleanups
including:
  - COMROOT/EXPDIR per case
  - ARCDIR and ATADIR directories for archive
  - STMP/RUNDIRS/${PSLOT}
- Added get config var from EXPDIR utility
[get_config_var.py](https://github.com/NOAA-EMC/global-workflow/pull/2961/files#diff-b780733ae1d45917730364e09a6c510e79d4cc8cffad6d9020c7961c53b987bc)
for getting config values in BASH

Resolves NOAA-EMC#2954
Resolves NOAA-EMC#3066
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA

Labels
CI/CD Issue related to CI/CD feature New feature or request
Projects
None yet
Milestone
No milestone
2 participants
@WalterKolczynski-NOAA @TerrenceMcGuinness-NOAA