Trying to understand RegridDataPlane followed by GridStat (in the same job) output

[robdarvell]

Hi,
I have been trying to improve the efficiency of running my gridstat task (which involves different grids and multiple fields) by splitting the grid aspect into running the regriddataplane task followed by the gridstat task. (I do the regrid on both the obs and fcst bits)
This has improved things, especially in my vector running, but I have run into a question which I have failed to understand what settings I have missed.

When running regriddataplane for multiple levels for the same field eg: (x/y)Winds @ 100hPa, 250hPa, 500hPa which are distinguished in the levels field (eg: “(@100,*,*)”), given the regrid job won’t place them into a similar structure in the output from regriddataplane I have included the output_field_name option in order to rename things. My name setting part of the config looks like:

FCST_REGRID_DATA_PLANE_VAR1_INPUT_FIELD_NAME = x_wind_0
FCST_REGRID_DATA_PLANE_VAR1_INPUT_LEVEL = "(@100,*,*)"
FCST_REGRID_DATA_PLANE_VAR1_OUTPUT_FIELD_NAME = x_wind_100
FCST_REGRID_DATA_PLANE_VAR2_INPUT_FIELD_NAME = y_wind_0
FCST_REGRID_DATA_PLANE_VAR2_INPUT_LEVEL = "(@100,*,*)"
FCST_REGRID_DATA_PLANE_VAR2_OUTPUT_FIELD_NAME = y_wind_100
FCST_REGRID_DATA_PLANE_VAR3_INPUT_FIELD_NAME = x_wind_0
FCST_REGRID_DATA_PLANE_VAR3_INPUT_LEVEL = "(@250,*,*)"
FCST_REGRID_DATA_PLANE_VAR3_OUTPUT_FIELD_NAME = x_wind_250
FCST_REGRID_DATA_PLANE_VAR4_INPUT_FIELD_NAME = y_wind_0
FCST_REGRID_DATA_PLANE_VAR4_INPUT_LEVEL = "(@250,*,*)"
FCST_REGRID_DATA_PLANE_VAR4_OUTPUT_FIELD_NAME = y_wind_250

Running this in regriddataplane does indeed rename the variables eg: x_wind_100, y_wind_100, x_wind_250 etc, with level setting as (*,*) in the newly created netcdf file.
I have then included those new names in my gridstat job. (and updated the file_type to NETCDF_MET)
(I don't know why, but below the levels are (star,star) (they just didn't seem to look like that just now when I hit return)

BOTH_VAR1_NAME = x_wind_100
BOTH_VAR1_LEVELS = "(*,*)"
BOTH_VAR1_OPTIONS = is_u_wind=true
BOTH_VAR2_NAME = y_wind_100
BOTH_VAR2_LEVELS = "(*,*)"
BOTH_VAR2_OPTIONS = is_v_wind=true
BOTH_VAR3_NAME = x_wind_250
BOTH_VAR3_LEVELS = "(*,*)"
BOTH_VAR3_OPTIONS = is_u_wind=true
BOTH_VAR4_NAME = y_wind_250
BOTH_VAR4_LEVELS = "(*,*)"
BOTH_VAR4_OPTIONS = is_v_wind=true

The gridstat job then runs through fine so the relevant fields it picks up the correct combinations to generate the stats. (I have also added the set_attr_field, set_attr_level in the options list to update the output file, but that doesn't make any difference to my next query)

However, my problem/question arises when it comes to trying to also include the vector stats. Adding the is_u_wind and is_v_wind, as I do when I just run the gridstat job I now get a lot of the following messages in the output:

**GridStatConfInfo::process_config() -> For U-wind, found multiple matching V-wind field array entries! Using the first match found. Set the "level" strings to differentiate between them.**

From looking through the code I see that this seems to be found in grid_stat_conf_info.cc (in the section from line 245).

It seems to be then selecting an incorrect combination when doing the vector stats (the first combination appears OK, VAR1 and VAR2 combined is OK). I am trying to work out how I would go about changing the levels string to differentiate between them as I have had to set (assuming I have done it correctly) them as (,) in the regriddataplane part. Also trying to understand what is involved in the field array entry in order for them to come back as matches even when the var?_name parts are different, Is there some other entry in the file which I am missing. (not getting far with my back reading through the c++)

I was trying to avoid needing to output each field level combination in the regrid job into a different file (and naming them somehow, couldn’t get that working when I did attempt it).

Any thoughts?
Many thanks
Rob

[JohnHalleyGotway]

@robdarvell, thanks for reaching out about this issue. I was able to replicate the behavior you describe and have some followup.

First though, let me offer a formatting tip for GitHub discussions. All discussions and issue comments are written in the Markdown lightweight markup language. Here's some Markdown formatting tips from GitHub. When typing code or commands, especially when including special characters like the @ symbol or the * character, it's best to place them inside backticks. For inline items, surround them with single backticks... like this... or for multi-line, use 3 backticks on a line to start it... then write commands on individual lines... and then end it with 3 backticks, like this:

echo "hello world"

I've edited your original post to include the use of single (`) backticks for inline code and triple (```) backticks for code blocks.

On to the issue at hand. I did some testing and have determined that we should refine/improve the level handling logic in the vx_data2d_nc_met library code. The warning message recommends that the level string be set to differentiate between the multiple matches... and I note that running with -v 3 includes more info about multiple matches:

DEBUG 3: U-wind field array entry 1 matches V-wind field array entry 2.
DEBUG 3: U-wind field array entry 1 matches V-wind field array entry 4.
WARNING: 
WARNING: GridStatConfInfo::process_config() -> For U-wind, found multiple matching V-wind field array entries! Using the first match found. Set the "level" strings to differentiate between them.
WARNING: 
DEBUG 3: V-wind field array entry 2 matches U-wind field array entry 1.
DEBUG 3: V-wind field array entry 2 matches U-wind field array entry 3.
WARNING: 
WARNING: GridStatConfInfo::process_config() -> For V-wind, found multiple matching U-wind field array entries! Using the first match found. Set the "level" strings to differentiate between them.
WARNING: 

I have found that while the vx_data2d_nc_met library requires that the level string be set to something, it DOES NOT actually matter how it is set! It just uses the NetCDF variable name to get the 2D data and essentially ignores the level setting. Instead, the rules should be...

1. Support the (*,*) notation.
2. Support the NetCDF level attribute convention... meaning that if NetCDF variable has a level attribute, allow it the level string in the config file to be set to it.
3. DO NOT match variables if the level string is not set as (1) or (2) above.

Additionally, we should update the U/V matching logic to avoid the multiple matches warning message shown above.

I'll write up a MET issue to clean up this logic and will note it here. Unfortunately, I don't have an easy fix for using MET version 12.0.2 to verify U/V with vector statistics (i.e. VL1L2, VAL1L2, and/or VCNT line types) at multiple levels when reading data from MET NetCDF files. As you've already pointed out, doing so with separate grid_stat runs for each vertical level would work but obviously is not the preferred solution.

[JohnHalleyGotway]

FYI, @robdarvell I documented this in MET issue dtcenter/MET#3087 and do have a proposed fix in pull request dtcenter/MET#3088. The issue is that when reading data from MET NetCDF files, all of the level strings were basically being hard-coded as (*,*) regardless of how they were actually set in the config file. This causes the "multiple matches" warning since all the level strings are the same.

The fix is to preserve the level strings that are set in the configuration file, so that the U's and V's can be matched up well using their level strings. Still need to carefully review the regression test output to make sure this change doesn't have negative, unintended consequences.

[JohnHalleyGotway]

The fix for this was just merged into MET's develop branch by dtcenter/MET#3088. It will be included in the next MET version 12.1.0 release. But you're welcome to checkout and compile MET's develop branch at any time to test if you'd like.