Getting MODE to work with self made data

[jcorner1]

I would like to note this is a bit of a time crunch, but I essentially have a file that is almost the exact same as (hard press to find the difference at this point from the working data) as the original data which is working fine in MODE. I have tried numerous different methods to try and get this data to work with MODE and feel that I am super close and might be missing something super basic or a minuet difference in the config, netcdf file, etc. As I stated before, this is a bit of a time crunch but if it can get working that would be great! I will send my data through the ftp server in the corner_data directory. Thanks!

[georgemccabe]

Hi @jcorner1, the large files you uploaded to the FTP are being corrupted again, likely due to the same issues we saw in #1924. If you are generating your own files that are meant to match the format of files you have previously read into MET, I would recommend running ncdump -h {filename} on both the file you created and the original to print the header information, pipe each result to a text file, then run the diff utility to compare the two text files. MET can read CF-compliant NetCDF files, so there may be some missing attributes that are required. If you are unable to track down the differences you could send us the text files that contain the header info. If you need to send the full files, you could try using Google Drive again, but please include the original file as well so I can compare it to your file.

We often recommend running plot_data_plane on a new file to see if it can be read by the other MET tools. You could run that with an increased verbosity to see if that gives you any other clues as to what is missing from the file. The command would look something like this:

plot_data_plane mrms_lcref202006050000.nc4 out.ps 'name="mrms_lcref"; level="(*,*)";' -v 5

You could also try increasing the verbosity of the MODE log output by setting LOG_MODE_VERBOSITY to an integer up to 10 (the default level is 2) in the METplus config file to see if you receive additional useful information in the logs.

Please note that many of our staff are still out of the office and others are preparing for a conference next week, so the support we can provided this week is limited, but we will do our best to help given the circumstances.

[jcorner1]

Hi @georgemccabe,

I have put all my files in this folder (https://drive.google.com/drive/folders/1diSpbv-0t3wNQEHwoBUqZV6FYsLngfZa?usp=share_link). I have tried matching the attributes the best as possible. I did get it to match but I don't think those data are uploaded in there. I'm just not sure what's giving it the issues.

[georgemccabe]

Hi @jcorner1, I only see 1 MRMS file in the folder, which I assume is the file you created yourself. Please also upload the original MRMS file that you are trying to mimic so I can compare them. Thanks.

[jcorner1]

Hi @georgemccabe,

I have placed the original mrms file in there. furthermore, I have changed the new one to be the more correct file although it basically changes very little

[georgemccabe]

Thanks for sending these files. The file ending with _OG could be read into MET because it has a global attribute to note the CF conventions:

// global attributes:
                :Conventions = "CF-1.6" ;

It looks like the file you created has this info as an attribute for the mrms_lcref field instead:

mrms_lcref:Conventions = "CF-1.6" ;

You can get around this by explicitly setting the file type with file_type=NETCDF_NCCF; to tell MET that the file should be interpreted as a CF-compliant NetCDF file.

/usr/local/met/bin/plot_data_plane mrms_lcref202006050000.nc4 out.ps 'name="mrms_lcref"; level="(*,*)"; file_type=NETCDF_NCCF;' -v 5

You can set this in your METplus config file like this:

OBS_VAR1_OPTIONS = file_type=NETCDF_NCCF;

The above Plot-Data-Plane command ran successfully, but the log output showed that the timing information was not read properly.

DEBUG 4: NcCfFile::open() -> could not extract valid time from the "time" variable.
DEBUG 4: NcCfFile::open() -> could not extract valid time from file name.
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file name.

...

DEBUG 4:    valid time: 19700101_000000
DEBUG 4:    lead time: 000000
DEBUG 4:    init time: 19700101_000000
DEBUG 4:    accum time: 000000

This means that MET will not be able to properly match the data with other files at the correct times. The _OG file you sent also did not include any timing information.

I found another MRMS file you had sent for #1924 named mrms_lcref2022-11-04T21_00_00.nc4 that does have timing information:

        int64 time ;
                time:long_name = "initial time of forecast" ;
                time:standard_name = "forecast_reference_time" ;
                time:units = "seconds since 1970-01-01" ;
                time:calendar = "proleptic_gregorian" ;

...

 time = 1559001600 ;

Adding this variable to your NetCDF file should fix this issue. It is possible to override the metadata from the input in MET (see this page of the documentation and search for "set_attr") to set the init or valid time, but I would recommend setting it properly in the file you are creating.

[jcorner1]

Hi @georgemccabe,

Sorry for the long wait for a reply... AMS takes up a lot of energy. Since then, I have gotten pretty close to fixing the issues I had before. I will put my data in the same google drive folder as I did previously (https://drive.google.com/drive/folders/1diSpbv-0t3wNQEHwoBUqZV6FYsLngfZa?usp=share_link). the original files have OG in them. Unable to find a difference in the files at this point except obviously the changed reflectivity data and HRRR raw output is a grib2 and the new is nc4. Let me know if there is any information MET is missing. I have attached the log and config file to help.

metplus.log.20230119220523.txt
MODE_HRRR_FCST_RADAR_OBS.txt

[georgemccabe]

Hi @jcorner1,

The MRMS data can be read into MET without issue except for the lack of time information that I mentioned in my response earlier.

It looks like it is interpreting the HRRR data as lat/lon instead of lambert conformal because the dimensions of the variables are (y,x) and not (lat, lon). The original grib2 data can be read into MET directly and the grid information will be read properly.

[jcorner1]

Hi @georgemccabe,

Focusing on the data I created for HRRR, I've tried a few different ways of creating the .nc4 file and appeared to get a different error in the log file which I would conclude to be a step in the right direction. The new error is

ERROR : StringArray::operator[](int) const -> range check error!

Let me know if there is a way to remedy this issue. Furthermore, my main issue with the HRRR data is I'm using copy to create a replica of the original file for the purpose of changing the reflectivity data and then running it through the MODE wrapper. My confusion is why this method does not work as I have worked every detail to be the same as the raw data but there is a different result. I've looked for other avenues or even scripts to create working data. If you potentially know of another route to go, please let me know. Thanks!

EDIT
I have uploaded the new data in the google drive (https://drive.google.com/drive/folders/1diSpbv-0t3wNQEHwoBUqZV6FYsLngfZa?usp=sharing)

[georgemccabe]

Hi @jcorner1, running your new file through plot_data_plane with verbosity set to 6 shows a number of log messages that identify missing information.

/usr/local/met/bin/plot_data_plane hrrr_t20200602T0600z_wrfsfcf18.nc4 out.ps 'name="refc"; level="(*,*)";' -v 6

DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file name.
DEBUG 4: NcCfFile::get_grid_from_dimensions() -> The coordinate variable "lat" does not exist.
DEBUG 4: NcCfFile::get_grid_from_dimensions() -> The coordinate variable "lon" does not exist.
DEBUG 4: NcCfFile::get_grid_from_dimensions() -> The coordinate variable "x" does not exist.
DEBUG 4: NcCfFile::get_grid_from_dimensions() -> The coordinate variable "y" does not exist.
DEBUG 6:
DEBUG 6: NcCfFile::get_grid_from_coordinates() -> collect GRID info from "hail".
DEBUG 6:
DEBUG 4:
DEBUG 4: NcCfFile::get_grid_from_coordinates() -> unknown units [1] for the coordinate variable [atmosphere]
DEBUG 4:
DEBUG 4:
DEBUG 4: NcCfFile::get_grid_from_coordinates() -> unknown units [hours] for the coordinate variable [step]
DEBUG 4:
ERROR :
ERROR : StringArray::operator const -> range check error!
ERROR :

Instead of using an external tool to copy and modify the data, I would recommend writing a Python Embedding script to read the grib data and make any modifications needed in the script. You can use this script to pass the data into the MET tools. Here are some resources:

Python Embedding for 2D Data from the MET User's Guide
MODE Python Embedding use case example
Example Python Embedding script that uses pygrib Python package
Example Python Embedding script that uses cfgrib Python package

EDIT: If necessary you can also use Python Embedding to pass in the data from the NetCDF file you created and explicitly set the attributes needed to describe the data to MET. However, I think it would be better to read the GRIB2 data directly into the Python Embedding script, perform any modifications you need, and pass the data to MET.

Also, this page contains many Python Embedding examples with sample data, scripts, and calls to plot_data_plane. We always recommend using plot_data_plane to generate an image to verify that MET is reading the data in the correct location on the grid.

[georgemccabe]

Hi @jcorner1, the ll values need to be the lower left corner. You will want to flip the data. This should work: met_data = met_data[::-1]

[jcorner1]

Hi @georgemccabe,

Thanks for the help so far. Pretty much almost done. I can run the data through MODE and get results. However inspection of the HRRR output from MET and the output from Matplotlib shows the actual values being about county south of MET is currently projecting it. Just to make sure I did compare the MRMS as well and there was no shift, so I think my input in attributes for HRRR would have to be incorrect at some point. I've looked it over a few times and can not seem to find anything that doesn't match. Is there a way to find the correct. Here are the attrs I have set which has a lot of the original left in:

attrs = {
        'valid': valid.strftime("%Y%m%d_%H%M%S"),
        'init':  init.strftime("%Y%m%d_%H%M%S"),
        'lead':  fhour,
        'accum': '00',
        'name': 'Reflectivity',
    
        'GRIB_name' : 'Maximum/Composite radar reflectivity',
        'GRIB_shortName' : 'refc',
        'GRIB_units' : 'dB',
        'long_name' : 'Maximum/Composite radar reflectivity',
        'units' : 'dB',
        'type' : 'Lambert Conformal',
        'level' : 'surface',
        
        'grid': {
            'GRIB_paramId' : 260390,
            'GRIB_dataType' : 'fc',
            'GRIB_numberOfPoints' : 1905141,
            'GRIB_typeOfLevel' : 'atmosphere',
            'GRIB_stepUnits' : 1,
            'GRIB_stepType' : 'instant',
            'GRIB_gridType' : 'lambert',
            'GRIB_DxInMetres' : 3000.0,
            'GRIB_DyInMetres' : 3000.0,
            'GRIB_LaDInDegrees' : 38.5,
            'GRIB_Latin1InDegrees' : 38.5,
            'GRIB_Latin2InDegrees' : 38.5,
            'GRIB_LoVInDegrees' : 262.5,
            'GRIB_NV' : 0,
            'GRIB_Nx' : 1799,
            'GRIB_Ny' : 1059,
            'GRIB_cfName' : 'unknown',
            'GRIB_cfVarName' : 'unknown',
            'GRIB_gridDefinitionDescription' : 'Lambert Conformal can be secant or tangent, conical or bipolar',
            'GRIB_iScansNegatively' : 0,
            'GRIB_jPointsAreConsecutive' : 0,
            'GRIB_jScansPositively' : 1,
            'GRIB_latitudeOfFirstGridPointInDegrees' : 21.138123,
            'GRIB_latitudeOfSouthernPoleInDegrees' : 0.0,
            'GRIB_longitudeOfFirstGridPointInDegrees' : 237.280472,
            'GRIB_longitudeOfSouthernPoleInDegrees' : 0.0,
            'GRIB_missingValue' : 9999,
            'name': 'Reflectivity',
            'type' : 'Lambert Conformal',
            'hemisphere' : 'N',
            'nx' : 1799,
            'ny' : 1059,
            'scale_lat_1' : 38.5,
            'scale_lat_2' : 38.5,
            'd_km' : 3.0, 
            #'r_km' : 6357.0,
            'r_km' : 6367.47,
            'lon_orient' : -95.0,
            'lat_pin' : 21.13812300000003,
            #'lat_pin' : 47.8386235,
            'lon_pin' : 237.28047200000003,
            #'lon_pin' : 225.90452027,
            'x_pin' : 0.0,
            'y_pin' : 0.0
            }
        }        
        

[georgemccabe]

I think you want to use the LoV value (262.5) as the lon_orient. Also, did the earth radius (r_km) value come from the file info? I noticed that was different than the example from the documentation (6371.2). The rest seem correct as far as I can tell. I would try adjusting those values and run through plot_data_plane to compare to the Matplotlib plots.

If you still can't get it to map properly, you could send an image of the matplotlib plot, the input file that was used, the Python Embedding script you are using, and the plot_data_plane command.

[jcorner1]

Hi @georgemccabe, that was it. I tried the LoV value at one point, but I put 265.0 instead of 262.5... they joys of coding and missing small yet simple details! Anyways, thanks so much for the help! I owe DTC a lot!!

[georgemccabe]

Great to hear! I am going to lock this discussion to prevent future posts. If you have any other questions or issues, please create a new discussion. Thanks!