Add tests to convert sat bias files and save them to R2D2; make calls to R2D2 more generic #37

CoryMartin-NOAA · 2022-04-01T21:12:30Z

Addresses the remainder of #7

Two new tests are added (but only if the full bundle is built)

test_gdasapp_convert_gsi_satbias will take the script added in Add driver script to convert GSI bias coefficient files for UFO #36 and run it with new sample data added
test_gdasapp_store_gsi_satbias will store the output of the above test into a test R2D2 database

Additionally, a generic call to R2D2 store has been added to ush/ufsda/r2d2.py, and changes have been made to the SOCA test to use this generic R2D2 store function.

guillaumevernieres

A few minor comments/questions. Looks good, thanks @CoryMartin-NOAA .

guillaumevernieres · 2022-04-01T21:31:12Z

test/atm/test_convert_gsi_satbias.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+set -ex


still need -x?

I thought it might be useful for debugging purposes

guillaumevernieres · 2022-04-01T21:35:07Z

test/atm/test_store_gsi_satbias.py

+                      'root': os.path.join(os.getcwd(), 'r2d2-test')}, },
+        'fetch_order': ['test'],
+        'store_order': ['test'],
+    }


couldn't that be a call to genyaml and we keep an r2d2_config.yaml template somewhere?

Sure, do you think that is what should be done in this and your existing test? Or stick with this approach for now? Does it make sense for the same r2d2 db to be used for both the atm and soca tests?

I think so, we have a few r2d2_config dictionaries defined here and there, it might be a good idea to consolidate how we deal with the config. I would say create an issue somewhere so we can remember to deal with it later, or change it now? Up to you. I won't block this pr for this issue.
As for the r2d2 db, yes, I do think we should have the same one.

guillaumevernieres · 2022-04-01T21:35:57Z

test/soca/create_obsdb.py

-                source_file=f'{source_dir}/{obs_type}_{year}{month}{day}.nc4',
-                ignore_missing=True,
-            )
+    ufsda.r2d2.store(config)


emcbot · 2022-04-01T21:55:31Z

Automated Pull Request Testing Results:
Machine: hera

Start: Fri Apr  1 21:15:18 UTC 2022 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Apr  1 21:55:13 UTC 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Fri Apr  1 21:55:31 UTC 2022
Tests: 89% tests passed, 1 tests failed out of 9
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/PR/37/GDASApp/build/log.ctest

emcbot · 2022-04-01T21:56:08Z

Automated Pull Request Testing Results:
Machine: orion

Start: Fri Apr  1 16:15:11 CDT 2022 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Apr  1 16:55:51 CDT 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Fri Apr  1 16:56:07 CDT 2022
Tests: 89% tests passed, 1 tests failed out of 9
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/PR/37/GDASApp/build/log.ctest

CoryMartin-NOAA · 2022-04-04T17:34:00Z

Build fails on Hera because da disk quota has been exceeded.

emcbot · 2022-04-04T18:30:02Z

Automated Pull Request Testing Results:
Machine: orion

Start: Mon Apr  4 12:45:25 CDT 2022 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Apr  4 13:28:59 CDT 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon Apr  4 13:30:01 CDT 2022
Tests: 89% tests passed, 1 tests failed out of 9
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/PR/37/GDASApp/build/log.ctest

emcbot · 2022-04-04T19:59:23Z

Automated Pull Request Testing Results:
Machine: orion

Start: Mon Apr  4 14:15:13 CDT 2022 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Apr  4 14:58:24 CDT 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon Apr  4 14:59:22 CDT 2022
Tests: 89% tests passed, 1 tests failed out of 9
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/PR/37/GDASApp/build/log.ctest

emcbot · 2022-04-04T20:59:30Z

Automated Pull Request Testing Results:
Machine: orion

Start: Mon Apr  4 15:15:15 CDT 2022 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Apr  4 15:58:41 CDT 2022
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Mon Apr  4 15:59:30 CDT 2022
Tests: 100% tests passed, 0 tests failed out of 9

guillaumevernieres

Looks good. Thanks @CoryMartin-NOAA .
I added a few comments that could be addressed in future pr's.

CoryMartin-NOAA added 14 commits April 1, 2022 11:54

Start of new unit tests

61797d3

Script to run satbias converter

eb622a1

incremental save

5286278

attempt to consolidate r2d2 obs stores

ab0f8de

coding norm attempt

c6ae6b4

fix pythonpath hopefully

c1559df

date to time

69dd87a

Fix typo

17a00c0

make r2d2 store very generic

1e094b5

Typo

9b4fe74

missing var

dd02f2b

Add in satbias store commands

e93f767

Fix for coding norms

153e263

Merge branch 'develop' into feature/satbias_r2d2_tests

e3d4856

CoryMartin-NOAA self-assigned this Apr 1, 2022

CoryMartin-NOAA requested a review from guillaumevernieres April 1, 2022 21:12

CoryMartin-NOAA added hera-RT Queue for automated testing on Hera orion-RT Queue for automated testing on Orion labels Apr 1, 2022

CoryMartin-NOAA requested a review from RussTreadon-NOAA April 1, 2022 21:12

emcbot added orion-RT-Running Automated testing running on Orion hera-RT-Running Automated testing running on Hera and removed orion-RT Queue for automated testing on Orion hera-RT Queue for automated testing on Hera labels Apr 1, 2022

guillaumevernieres previously approved these changes Apr 1, 2022

View reviewed changes

emcbot added hera-RT-Failed Automated testing failed on Hera and removed hera-RT-Running Automated testing running on Hera labels Apr 1, 2022

emcbot added orion-RT-Failed Automated testing failed on Orion and removed orion-RT-Running Automated testing running on Orion labels Apr 1, 2022

CoryMartin-NOAA added orion-RT Queue for automated testing on Orion and removed hera-RT-Failed Automated testing failed on Hera orion-RT-Running Automated testing running on Orion labels Apr 4, 2022

emcbot added orion-RT-Running Automated testing running on Orion and removed orion-RT Queue for automated testing on Orion labels Apr 4, 2022

CoryMartin-NOAA requested a review from guillaumevernieres April 4, 2022 18:27

emcbot added orion-RT-Failed Automated testing failed on Orion and removed orion-RT-Running Automated testing running on Orion labels Apr 4, 2022

Remove time_window from bc store

c0488cb

CoryMartin-NOAA added orion-RT Queue for automated testing on Orion and removed orion-RT-Failed Automated testing failed on Orion labels Apr 4, 2022

emcbot added orion-RT-Running Automated testing running on Orion and removed orion-RT Queue for automated testing on Orion labels Apr 4, 2022

emcbot added orion-RT-Failed Automated testing failed on Orion and removed orion-RT-Running Automated testing running on Orion labels Apr 4, 2022

bugfixes hopefully

8b6ef51

CoryMartin-NOAA added orion-RT Queue for automated testing on Orion and removed orion-RT-Failed Automated testing failed on Orion labels Apr 4, 2022

emcbot added orion-RT-Running Automated testing running on Orion and removed orion-RT Queue for automated testing on Orion labels Apr 4, 2022

emcbot added orion-RT-Passed Automated testing successful on Orion and removed orion-RT-Running Automated testing running on Orion labels Apr 4, 2022

guillaumevernieres approved these changes Apr 4, 2022

View reviewed changes

CoryMartin-NOAA merged commit 52b8c76 into develop Apr 5, 2022

CoryMartin-NOAA deleted the feature/satbias_r2d2_tests branch April 5, 2022 12:25

CoryMartin-NOAA mentioned this pull request Apr 5, 2022

Add in scripts to convert and save to R2D2 sat bias files #7

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tests to convert sat bias files and save them to R2D2; make calls to R2D2 more generic #37

Add tests to convert sat bias files and save them to R2D2; make calls to R2D2 more generic #37

CoryMartin-NOAA commented Apr 1, 2022

guillaumevernieres left a comment

guillaumevernieres Apr 1, 2022

CoryMartin-NOAA Apr 4, 2022

guillaumevernieres Apr 1, 2022

CoryMartin-NOAA Apr 4, 2022

guillaumevernieres Apr 4, 2022

guillaumevernieres Apr 1, 2022

emcbot commented Apr 1, 2022

emcbot commented Apr 1, 2022

CoryMartin-NOAA commented Apr 4, 2022

emcbot commented Apr 4, 2022

emcbot commented Apr 4, 2022

emcbot commented Apr 4, 2022

guillaumevernieres left a comment

Add tests to convert sat bias files and save them to R2D2; make calls to R2D2 more generic #37

Add tests to convert sat bias files and save them to R2D2; make calls to R2D2 more generic #37

Conversation

CoryMartin-NOAA commented Apr 1, 2022

guillaumevernieres left a comment

Choose a reason for hiding this comment

guillaumevernieres Apr 1, 2022

Choose a reason for hiding this comment

CoryMartin-NOAA Apr 4, 2022

Choose a reason for hiding this comment

guillaumevernieres Apr 1, 2022

Choose a reason for hiding this comment

CoryMartin-NOAA Apr 4, 2022

Choose a reason for hiding this comment

guillaumevernieres Apr 4, 2022

Choose a reason for hiding this comment

guillaumevernieres Apr 1, 2022

Choose a reason for hiding this comment

emcbot commented Apr 1, 2022

emcbot commented Apr 1, 2022

CoryMartin-NOAA commented Apr 4, 2022

emcbot commented Apr 4, 2022

emcbot commented Apr 4, 2022

emcbot commented Apr 4, 2022

guillaumevernieres left a comment

Choose a reason for hiding this comment