Trouble reading ERA5 data

[hankenstein2]

I am attempting to use grid_stat to read in GFS as the forecast data and ERA5 as the obs. I am getting the error message.

Trouble reading observation file "precipitation_amount_1hour_Accumulation.nc"

I am not sure if it is the config file that I have filled out properly or the data is not in a very usable format.
The command line string I used is below.
The data is in /d1/personal/fisherh/Prototype7 on seneca which was pulled directly from the NOAA Open DataDdissemination program

https://www.noaa.gov/information-technology/open-data-dissemination

grid_stat gfs.t00z.pgrb2.0p25.f000 precipitation_amount_1hour_Accumulation.nc GridStatConfig_default

[CPKalb]

Hi Hank:

MET only reads netCDF files that are either CF compliant, in MET's netCDF format, or in the format produced from wrf_interp. This file is not in any of these formats, which is why the error is occurring.

Although it's technically not CF compliant, it has enough of the attributes of that format that I can tell MET to read it as if it were a CF compliant file. To do this, you need to add file_type=NETCDF_NCCF; to the configuration. I've tested this in plot data plane using the following:

plot_data_plane /d1/personal/fisherh/Prototype7/precipitation_amount_1hour_Accumulation.nc test_precip.ps
'name="precipitation_amount_1hour_Accumulation"; level="(20180301,*,*)"; file_type=NETCDF_NCCF;'

When I run plot data plane this way, I do get the following information about the data.
DEBUG 2: NcCfFile::getData(NcVar *, const LongArray &, DataPlane &) const -> data was flipped to north.

However, comparing the output from plot data plane to ncview (shown below), I can verify that the data was read in correctly. I had to flip the color scale in ncview to be able to see the lowest contour levels.

Try add the following to your configuration file in the obs dictionary and let me know if this fixes the issue.
file_type = NETCDF_NCCF;

One other important thing to note is that your precipitation is in meters. I couldn't find APCP in the grib file. But, there may be a units conversion necessary. Which field are you using from the grib file?

Tina

[hankenstein2]

Thanks Tina! So I did/do have this in the config file I am using but it does not seem to be working. As for APCP in the GFS file, oops! Of course the 00 lead time won't have accumulated precip so I will switch to the 06.

obs = {

field= [
{
name = "precipitation_amount_1hour_Accumulation";
level = [ "(@20180301,,)" ];
file_type = NETCDF_NCCF;
}
];
}

[CPKalb]

Hi Hank:
Are the observations 1 hour accumulations? Ideally we would want to match the model accumulation to the observations. I don't see a 1 hour accumulation, only two 6 hour accumulations in the 6 hour file.

Try adding the file type outside of the field area such as:
obs = {
file_type = NETCDF_NCCF;
field= [
{
name = "precipitation_amount_1hour_Accumulation";
level = [ "(@20180301,*,*)" ];
}
];
}

When I do that, I get a complaint about the thresholds:
ERROR : GridStatVxOpt::process_config() -> The number of thresholds for each field in "fcst.cat_thresh" must match the number of thresholds for each field in "obs.cat_thresh".
I then added thresholds to the obs dictionary to look like this
obs = {
file_type = NETCDF_NCCF;

field= [
{
name = "precipitation_amount_1hour_Accumulation";
level = [ "(@20180301,*,*)" ];
cat_thresh = [ >0.0, >=5.0 ];
}
];
}

Trying to run this way produces another error:
WARNING:
WARNING: MetGrib2DataFile::data_plane() -> No matching record found for 'APCP/A03'
WARNING:
WARNING:
WARNING: read_data_plane() -> APCP/A03 not found in file: /d1/personal/fisherh/Prototype7/gfs.t00z.pgrb2.0p25.f006
WARNING:
ERROR :
ERROR : process_scores() -> no requested data found! Exiting...

Here, I switched to the 6 hourly accumulation just to see if I can get grid stat to run, by changing the level in the fcst variable (level = ["A06"];). It correctly warns me that there are 2 fields in the grib file that are 6 hourly accumulations. However, I also get another error:
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) -> the requested time 19700822_133821 for "precipitation_amount_1hour_Accumulation" variable does not exist (20180301_000000 and 20180331_230000).
WARNING:
WARNING:
WARNING: read_data_plane() -> precipitation_amount_1hour_Accumulation(@20180301,*,*) not found in file: /d1/personal/fisherh/Prototype7/precipitation_amount_1hour_Accumulation.nc
WARNING:
ERROR :
ERROR : process_scores() -> no requested data found! Exiting...
ERROR :

Here, I gave it the full time stamp including HHMMSS, by changing the obs level to be level = [ "(@20180301_060000,*,*)" ]'

And finally grid stat runs :-)!

We probably need to run pcp combine to convert the observations to be a 6 hourly accumulation to match the forecast. Ideally we would also want to set GRIB2_ipdtmpl_index" and GRIB2_ipdtmpl_val to discern between the two model fields. But, the grib numbers of the two fields are identical, so it won't help in this case.
WARNING: record 596 field 1: ipdtmpl[29] = 1, 8, 2, 0, 96, 0, 0, 1, 0, 1, 0, 0, 255, 0, 0, 2018, 3, 1, 6, 0, 0, 1, 0, 1, 2, 1, 6, 255, 0
WARNING: record 597 field 1: ipdtmpl[29] = 1, 8, 2, 0, 96, 0, 0, 1, 0, 1, 0, 0, 255, 0, 0, 2018, 3, 1, 6, 0, 0, 1, 0, 1, 2, 1, 6, 255, 0

I'm guessing the fields are actually exactly the same. If I look at the other model lead times, there is a running accumulation and a 6 hourly accumulation typically in the model files (for example, the 24 hour forecast has a 24 hour accumulation and a 6 hourly accumulation). At 6 hours, the running accumulation and 6 hourly accumulation are both 6 hourly. So, in this case it really doesn't matter which field MET uses.

Here is the configuration file:
GridStatConfig_ERA5.txt
Please let me know if this works.

Tina

[hankenstein2]

Amazing Tina! Thank you. I think we can close this one. Hank

…

On Wed, Apr 19, 2023, 1:48 PM Christina Kalb ***@***.***> wrote: Hi Hank: Are the observations 1 hour accumulations? Ideally we would want to match the model accumulation to the observations. I don't see a 1 hour accumulation, only a 6 hour in the 6 hour file. Try adding the file type outside of the field area such as: obs = { file_type = NETCDF_NCCF; field= [ { name = "precipitation_amount_1hour_Accumulation"; level = [ ***@***.***, *, *)" ]; } ]; } When I do that, I get a complaint about the thresholds: ERROR : GridStatVxOpt::process_config() -> The number of thresholds for each field in "fcst.cat_thresh" must match the number of thresholds for each field in "obs.cat_thresh". I then added thresholds to the obs dictionary to look like this obs = { file_type = NETCDF_NCCF; field= [ { name = "precipitation_amount_1hour_Accumulation"; level = [ ***@***.***,*,*)" ]; cat_thresh = [ >0.0, >=5.0 ]; } ]; } Trying to run this way produces another error: WARNING: WARNING: MetGrib2DataFile::data_plane() -> No matching record found for 'APCP/A03' WARNING: WARNING: WARNING: read_data_plane() -> APCP/A03 not found in file: /d1/personal/fisherh/Prototype7/gfs.t00z.pgrb2.0p25.f006 WARNING: ERROR : ERROR : process_scores() -> no requested data found! Exiting... Here, I tried switching to the 6 hourly accumulation just to see if I can get grid stat to run, by changing the level in the fcst variable (level = ["A06"];). It correctly warns me that there are 2 fields in the grib file that are 6 hourly accumulations. However, I also get another error: WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) -> the requested time 19700822_133821 for "precipitation_amount_1hour_Accumulation" variable does not exist (20180301_000000 and 20180331_230000). WARNING: WARNING: WARNING: read_data_plane() -> ***@***.***,*,*) not found in file: /d1/personal/fisherh/Prototype7/precipitation_amount_1hour_Accumulation.nc WARNING: ERROR : ERROR : process_scores() -> no requested data found! Exiting... ERROR : Here, I tried giving it the full time stamp including HHMMSS, by changing the obs level to be level = [ ***@***.***_060000, *, *)" ]' And finally grid stat runs :-)! We probably need to run pcp combine to convert the observations to be a 6 hourly accumulation to match the forecast. Tina — Reply to this email directly, view it on GitHub <#2131 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHAUIKHBYUJSXVCMDOHJOP3XCBFQXANCNFSM6AAAAAAXEO7SVY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[CPKalb]

Okay, great! I'll close the discussion!

Tina