Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update to xgrid as default #330

Merged
merged 1 commit into from
Jan 3, 2023
Merged

update to xgrid as default #330

merged 1 commit into from
Jan 3, 2023

Conversation

jedwards4b
Copy link
Collaborator

Description of changes

Changes default atm ocn flux grid to xgrid.

Specific notes

Contributors other than yourself, if any:

CMEPS Issues Fixed (include github issue #):

Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial)

Any User Interface Changes (namelist or namelist defaults changes)?

Testing performed

Testing performed if application target is CESM:

  • (recommended) CIME_DRIVER=nuopc scripts_regression_tests.py
    • machines:
    • details (e.g. failed tests):
  • (recommended) CESM testlist_drv.xml
    • machines and compilers:
    • details (e.g. failed tests):
  • (optional) CESM prealpha test
    • machines and compilers
    • details (e.g. failed tests):
  • (other) please described in detail
    • machines and compilers
    • details (e.g. failed tests):

Testing performed if application target is UFS-coupled:

  • (recommended) UFS-coupled testing
    • description:
    • details (e.g. failed tests):

Testing performed if application target is UFS-HAFS:

  • (recommended) UFS-HAFS testing
    • description:
    • details (e.g. failed tests):

Hashes used for testing:

  • CESM:
  • UFS-coupled, then umbrella repostiory to check out and associated hash:
    • repository to check out:
    • branch/hash:
  • UFS-HAFS, then umbrella repostiory to check out and associated hash:
    • repository to check out:
    • branch/hash:

@jedwards4b jedwards4b self-assigned this Dec 15, 2022
@klindsay28
Copy link

@alperaltuntas, have we vetted aoflux_grid=xgrid in G compsets?
I looked over my email and only found comparisons in B compsets.

@jedwards4b
Copy link
Collaborator Author

@klindsay28 thank you for bringing this up, we are planning to make this the default grid with this PR and want to know ASAP if there is any reason for concern.

@gustavo-marques
Copy link
Collaborator

Thanks, @klindsay28 and @jedwards4b. What do we need to update (only CMEPS?) to be able to test the latest improvements in aoflux_grid. We will run a G case over the weekend and get back to you early next week.

@jedwards4b
Copy link
Collaborator Author

@gustavo-marques You can use this cmeps branch, or you can just use the cesm2_3_beta10 and set
aoflux_grid = 'xgrid' in user_nl_cpl

@gustavo-marques
Copy link
Collaborator

I downloaded cesm2_3_beta10, build a GMOM and got the following error after submitting the job:

Submitting job script qsub -q share  -l walltime=12:00:00  -A P93300612  -W depend=afterok:7739229.chadmin1.ib0.cheyenne.ucar.edu -v ARGS_FOR_SCRIPT='--resubmit' case.st_archive
ERROR: Command: 'qsub -q share  -l walltime=12:00:00  -A P93300612  -W depend=afterok:7739229.chadmin1.ib0.cheyenne.ucar.edu -v ARGS_FOR_SCRIPT='--resubmit' case.st_archive' failed with error 'qsub: 
ERROR: your job has been rejected.

your declared wallclock time (43200 seconds)
exceeds your maximum limit of 21600 seconds
the queue limit is 21600 seconds

Please contact [email protected]
for assistance' from dir '/glade/work/gmarques/cesm.cases/G/gmom.e23.GJRAv4.TL319_t061_zstar_N65.xgrid'

@jedwards4b
Copy link
Collaborator Author

It got the wallclock time wrong for the archiver, I'll look into that. You can work around the issue by setting
DOUT_S=FALSE.

@gustavo-marques
Copy link
Collaborator

There seems to be another problem with this sandbox (cesm2_3_beta10). The simulation run for 12 years, restarting every 6 years. The first restart was successful. On the second restart (year 13), I get the following error:

ERROR: CONTINUE_RUN is true but this case does not appear to have restart files staged in /glade/scratch/gmarques/gmom.e23.GJRAv4.TL319_t061_zstar_N65.xgrid/run rpointer.cpl_0001

@jedwards4b
Copy link
Collaborator Author

jedwards4b commented Dec 17, 2022 via email

@gustavo-marques
Copy link
Collaborator

Change MULTI_DRIVER did not work.

./xmlchange MULTI_DRIVER=False
ERROR: Did not find False in valid values for MULTI_DRIVER: ['TRUE']

@jedwards4b
Copy link
Collaborator Author

jedwards4b commented Dec 18, 2022 via email

@gustavo-marques
Copy link
Collaborator

Doing that also give errors. After adding FALSE to the allowed values, I did "./xmlchange MULTI_DRIVER=FALSE" and "./case.setup --reset".

Then, "./case.submit" gives

Traceback (most recent call last):
  File "/glade/work/gmarques/cesm.cases/G/gmom.e23.GJRAv4.TL319_t061_zstar_N65.xgrid/./case.submit", line 196, in <module>
    _main_func(__doc__)
  File "/glade/work/gmarques/cesm.cases/G/gmom.e23.GJRAv4.TL319_t061_zstar_N65.xgrid/./case.submit", line 179, in _main_func
    case.submit(
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/cime/CIME/case/case_submit.py", line 272, in submit
    run_and_log_case_status(
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/cime/CIME/utils.py", line 2433, in run_and_log_case_status
    rv = func()
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/cime/CIME/case/case_submit.py", line 257, in <lambda>
    functor = lambda: _submit(
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/cime/CIME/case/case_submit.py", line 166, in _submit
    case.check_case(skip_pnl=skip_pnl, chksum=chksum)
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/cime/CIME/case/case_submit.py", line 292, in check_case
    self.create_namelists()  # Must be called before check_all_input_data
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/cime/CIME/case/preview_namelists.py", line 90, in create_namelists
    import_and_run_sub_or_cmd(
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/cime/CIME/utils.py", line 644, in import_and_run_sub_or_cmd
    run_sub_or_cmd(cmd, cmdargs, subname, subargs, logfile, case, from_dir, timeout)
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/cime/CIME/utils.py", line 685, in run_sub_or_cmd
    getattr(mod, subname)(*subargs)
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/components/cmeps/cime_config/buildnml", line 571, in buildnml
    _create_drv_namelists(case, infile, confdir, nmlgen, files)
  File "/glade/work/gmarques/cesm.sandboxes/cesm2_3_beta10/components/cmeps/cime_config/buildnml", line 327, in _create_drv_namelists
    if maxinst == 1 and model != 'cpl' and not multi_driver:
UnboundLocalError: local variable 'multi_driver' referenced before assignment

@jedwards4b
Copy link
Collaborator Author

jedwards4b commented Dec 18, 2022 •
edited
Loading

Okay - one more change. In file components/cmeps/cime_config/buildnml

diff --git a/cime_config/buildnml b/cime_config/buildnml
index acaac4d..80fd28f 100755
--- a/cime_config/buildnml
+++ b/cime_config/buildnml
@@ -325,9 +325,7 @@ def _create_drv_namelists(case, infile, confdir, nmlgen, files):
     #if we are in multi-coupler mode the number of instances of mediator will be the max
     # of any NINST_* value
     maxinst = 1
-    if case.get_value("MULTI_DRIVER"):
-        maxinst = case.get_value("NINST_MAX")
-        multi_driver = True
+
     with open(nuopc_config_file, 'a', encoding="utf-8") as conffile:
         nmlgen.write_nuopc_config_file(conffile, data_list_path=data_list_path)
 
@@ -342,10 +340,7 @@ def _create_drv_namelists(case, infile, confdir, nmlgen, files):
                 newgroup = model.upper()+"_modelio"
             nmlgen.rename_group("modelio", newgroup)
 
-            if maxinst == 1 and model != 'cpl' and not multi_driver:
-                inst_count = case.get_value("NINST_" + model.upper())
-            else:
-                inst_count = maxinst
+            inst_count = maxinst
             if not model == 'drv':
                 for entry in ["pio_async_interface", 
                               "pio_netcdf_format",

Sorry this has been such a hassle.

@gustavo-marques
Copy link
Collaborator

Thanks! it worked now.

@jedwards4b
Copy link
Collaborator Author

@gustavo-marques any updates on your run?

@gustavo-marques
Copy link
Collaborator

The run will be done tomorrow. I will run the diagnostics and get back to you by the end of the day.

@gustavo-marques
Copy link
Collaborator

aoflux_grid = 'xgrid' gives very similar results with respect to the control in a G case (aoflux_grid = 'ogrid'). This gives us confidence to go ahead with xgrid as the default.

Below is the global mean ocean temperature in the two runs (ctrl and xgrid):
image

For reference, a full set of diagnostics comparing these two runs is available here.

@jedwards4b
Copy link
Collaborator Author

Thank you!

@jedwards4b jedwards4b merged commit 39b2b11 into ESCOMP:master Jan 3, 2023
@jedwards4b jedwards4b deleted the change_default_ocnatm_grid branch January 3, 2023 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fischer-ncar fischer-ncar Awaiting requested review from fischer-ncar

Assignees

@jedwards4b jedwards4b

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jedwards4b @klindsay28 @gustavo-marques