Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Patches to S2S scripts for Nov 2021 Air Force Deliverable #926

Merged
merged 80 commits into from
Nov 15, 2021
Merged

Patches to S2S scripts for Nov 2021 Air Force Deliverable #926

merged 80 commits into from
Nov 15, 2021

Conversation

emkemp
Copy link
Contributor

@emkemp emkemp commented Nov 9, 2021

Description

Numerous changes to S2S scripts for November 2021 Air Force deliverable, including replacing most shell scripts with Python 3.

emkemp and others added 30 commits October 14, 2021 09:05
@emkemp
Added script to generate GeoTIFF files from netCDF metrics data.
cfcbfe7
@karsenau
Updates to LIS forecast readers to add more models
efc4eff 
 Code updates to the LIS enesmble forecast reader and the
  precipitation forecast reader to include additional
  model entries (e.g., CFSv2).  Default entry is for "GEOS5".
@karsenau
Update LIS config documentation
10eff3f 
 - Added new lis config file entry and option documentation for
    the generic ensemble-based forecast forcing reader in LIS.

 - Added new entries and documentation in the lis config doc
    for the precipitation-only ensemble forecast forcing reader.
    (This information was missing before in the documentation)
@emkemp
Merge remote-tracking branch 'upstream/support/lisf-557ww-7.4' into f…
a169448 
…ix/lisf74-s2s-patches

Catching-up with support branch.
@karsenau
Merge remote-tracking branch 'emkemp/fix/lisf74-s2s-patches' into fix…
ac35814 
…/lisf74-s2s-patches
@emkemp
Updated s2smetrics scripts to comply with pylint and to be more flexi…
9e6cfbe 
…ble.
@emkemp
New batch job script. Added touch files.
9021a4b 
1. run_generate_metrics.sh is the new batch job script, and it accepts the
path to the s2smetrics/lib_bcsd_metrics directory as an argument.
2. Trimmed print statements for back-end Python scripts.  Also added creation
of "touch" files to mark when metric jobs are completed.  Touch files are in
same directory as the metrics netCDF files.
@emkemp
Added Python script to replace shell script.
2313893
@emkemp
Consolidation of s2spost output now command-line argument.
92e1dff 
Updated paths for testing on Discover with multiple NMME-forced LIS forecasts.
@emkemp
Added back-end CF and GeoTIFF postprocessing.
e4a5d51
@emkemp
Modified scripts to process all NMME LIS runs.
6863ef2
@emkemp
Addressed pylint objections.
af78b1b 
NOTE:  These scripts currently process only GEOSv2 LIS runs (for testing
efficiency).
@emkemp
Uses new conda environment on Discover. Fixes pylint string complaints.
d17db31
@emkemp
Fixed bug introduced when adding f-strings.
ddc0eed
@emkemp
Fixed filenames for TIFF files.
79b20b8
@emkemp
Revised GeoTIFF code to calculate median across entire ensemble (mult…
0e747c2 
…i-model).
@emkemp
Revised to use s2spost config file.
b4ce67d
@emkemp
Revised to use s2smetric config file.
7599b5b
@emkemp
Restored all NMME models.
1fc5de4
@emkemp
Cleaned up code.
0f8e3fd
@emkemp
Cleaned up code.
f39c33c
@emkemp
Cleaned up code.
9bfdde0
@emkemp
Cleaned up code.
73a89b0
@emkemp
Cleaned up code.
c2b4ed3
@emkemp
Cleaned up code.
38b1326
@emkemp
Cleaned up code.
6dace87
@emkemp
Cleaned up code.
be67d02
@emkemp
Cleaned up code.
803221f
@karsenau
Merge remote-tracking branch 'emkemp/fix/lisf74-s2s-patches' into fix…
e222cd4 
…/lisf74-s2s-patches
@emkemp
Addressed recent objections from pylint.
2846fa3 
* Replaced classic string formats with f-strings.
* Added encoding for open statements.
@jvgeiger
Copy link
Contributor

jvgeiger commented Nov 9, 2021

Thanks.

@jvgeiger
Copy link
Contributor

  1. I would like to remove the #SBATCH --mail-user lines from the scripts. I realize that this is a placeholder for the user to fill in, but if the user does not then
  • #SBATCH --mail-user= will end up not sending an email notification
  • #SBATCH [email protected] will end up sending an email to email.com, which is a registered domain and which most likely will bound back to SLURM.

Setting #SBATCH --mail-type=END is sufficient to tell SLURM to send an email notification to the submitter upon completion of the job.

  1. The lvt/utils/usaf/s2smetric/README files refers to "8u" scripts. The document calls out 8 scripts, but there are 9 in that directory - make_s2s_metric_geotiff.py is not referred to. Should I correct this to 8 or 9?

@karsenau
Copy link
Contributor

Thanks @jvgeiger!

@karsenau can take on #1, if @emkemp wants to do #2

@emkemp
Copy link
Contributor Author

emkemp commented Nov 10, 2021

I'll tackle the README file. (Sorry about that!)

@emkemp
Copy link
Contributor Author

emkemp commented Nov 10, 2021

I moved make_s2s_metric_geotiff.py into a separate directory. It is only in the repository in case we really need to create separate GeoTIFFs for each metric ensemble member (no sense throwing it away). But is it not considered part of s2smetric at this time.

The README has been updated.

@emkemp
Copy link
Contributor Author

emkemp commented Nov 10, 2021

I removed the --mail-user= in the script for the Bratseth retuning software (separate from S2S, won't interfere with the updates from @karsenau)

@karsenau
Copy link
Contributor

Thanks @emkemp !

I'm just about to commit the other removed "mail" lines in the slurm scripts.

@karsenau
Copy link
Contributor

@jvgeiger and @emkemp:

I realized the following script:
generate_s2safricom_da_config_scriptfiles_fcst.py

Has sort of a "misleading" naming convention, as this is the script that generates the forecast run, not the da run. If it's ok with you all, I would like to modify the script name and update some of the descriptions in the script.

The file currently exists in:
lis/configs/557WW-7.4-FOC/S2S_AFRICOM/generate_s2safricom_da_config_scriptfiles_fcst.py

Thanks!

@emkemp
Copy link
Contributor Author

emkemp commented Nov 10, 2021

Discover is going down at 9 am for maintenance, so you may be out of time. I would push your mail-user changes now.

@karsenau
Remove all mail-user entries for S2S slurm scripts
ecdc8a6 
- Removed remaining --mail-user entries in BCSD and other S2S
   related scripts.
- Modified name of the LIS forecast python script
- Updated documentation
@karsenau
Copy link
Contributor

@emkemp : Commit pushed over

@emkemp
Copy link
Contributor Author

emkemp commented Nov 10, 2021

Got it! @jvgeiger I think the fixes are all in.

razamora-nasa and others added 8 commits November 11, 2021 17:10
@jvgeiger jvgeiger merged commit b8ef488 into NASA-LIS:support/lisf-557ww-7.4 Nov 15, 2021
@emkemp emkemp deleted the fix/lisf74-s2s-patches branch May 17, 2022 19:15
zdenkoheyvaert pushed a commit to zdenkoheyvaert/LISF that referenced this pull request Dec 9, 2023
@jvgeiger
Merge pull request NASA-LIS#926 from emkemp/fix/lisf74-s2s-patches
6b4deaa 
Patches to S2S scripts for Nov 2021 Air Force Deliverable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jvgeiger jvgeiger

Labels
BugFix Fixes bug found in code HighPriority LISF 557WW 7.4
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@emkemp @jvgeiger @karsenau @razamora-nasa