NetCDF File Issue with MODE #2333

balbrightwpc · 2023-08-31T15:54:41Z

balbrightwpc
Aug 31, 2023

Hello myself and @ebower084 are working on generating a QPF ensemble mean netCDF file and then running that through MODE. When trying to run the netCDF file through MODE I get the following error:

ERROR :
ERROR : MetNcFile::data(NcVar *, const LongArray &, DataPlane &) const -> star found in bad slot
ERROR :
HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0:
#000: H5T.c line 1716 in H5Tclose(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
NetCDF: HDF error
file: ncFile.cpp line:33
HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0:
#000: H5T.c line 1716 in H5Tclose(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
NetCDF: HDF error
file: ncFile.cpp line:33

@ebower084 has tried changing some of the details in the netCDF file to match files that are confirmed to work with MODE but we keep running into this error. At this point we don't know if it is a netCDF file issue or if something needs to be adjusted in my MODE script for it to properly read the file. The ncdump for the file we are trying to use is the following:

dimensions:
x = 1059 ;
y = 1799 ;
variables:
double lat(x, y) ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:standard_name = "latitude" ;
double lon(x, y) ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:standard_name = "longitude" ;
double apcp_24hr(x, y) ;
apcp_24hr:FillValue = -9999. ;
apcp_24hr:long_name = "APCP" ;
apcp_24hr:level = "SURFACE" ;
apcp_24hr:units = "inches" ;

// global attributes:
:FileOrigins = "File generated 2023-08-30 18:24:17.692806 UTC via Python" ;
:MET_version = "V9.0.1" ;
:MET_tool = "None" ;
:Projection = "Lambert Conformal" ;
:file_type = "NETCDF_NCCF" ;
:hemisphere = "N" ;
:scale_lat_1 = "38.500000" ;
:scale_lat_2 = "38.500000" ;
:lat_pin = "21.138123" ;
:lon_pin = "-122.719528" ;
:x_pin = "0.000000" ;
:y_pin = "0.000000" ;
:lon_orient = "-97.500000" ;
:d_km = "3.000000" ;
:r_km = "6371.200000" ;
:nx = "1799" ;
:ny = "1059 grid_points" ;
:_NCProperties = "version=2,netcdf=4.8.1,hdf5=1.12.1" ;

If you need the actual netCDF file, let me know and I will work on sending that over to the anonymous FTP.

Thank you for any help you can provide.

Ben Albright

Answered by CPKalb

Sep 5, 2023

Hi Ben:

The problem is with your forecast file. It's not in a format that MET can read, although it's close to MET's netCDF format. There are a couple of issues to fix. The first is the global attribute file_type = "NETCDF_NCCF" ;. NETCDF_NCCF specifically refers to CF compliant netCDF files. Since yours is not in this format, it would be good to remove this attribute. Second, the dimensions are inconsistent between the global attributes and the variables. The global attributes list nx = 1799 and nx = 1059. However, at the top of the file, the dimensions say x = 1059 and y = 1799. Additionally, a MET NETCDF file typically uses lat and lon as the dimensions rather than x and y. I ran a qui…

View full answer

j-opatz · 2023-08-31T18:40:23Z

j-opatz
Aug 31, 2023
Collaborator

Hi Ben,

And thank you for your question. Given the wording of your error code,

ERROR :
ERROR : MetNcFile::data(NcVar *, const LongArray &, DataPlane &) const -> star found in bad slot
ERROR :

It seems like you may be improperly using the "*" character in your configuration file. From the MET User's Guide, it provides the following guidance for netCDF levels:

(i,…,j,*,*) for a single field, where i,…,j specifies fixed dimension values and *,* specifies the two dimensions for the gridded field.

So the only places in the level information that need an asterisk character are those 2-D arrays. In a MET configuration file, the dictionary might look like

field = [
     {
       name       = "apcp_24hr";
       level      = "(*,*)";
     }
];

Would you be able to provide the configuration file that is pulling in the netCDF information?

9 replies

balbrightwpc Sep 5, 2023
Author

Setting both levels to "( * , * )" I get the following error:

ERROR :
ERROR : MetNcFile::data(NcVar *, const LongArray &, DataPlane &) const -> star found in bad slot
ERROR :

I am trying to send the data but am having issues logging into the FTP. I've logged in the following way:

ftp -p ftp.rap.ucar.edu
username=anonymous

Then at the password prompt, I attempt to enter my email address and it just says the login is incorrect:

Password:
530 Login incorrect.
Login failed.

I've also tried adding the (-) as the first character of the password like the instructions say but I got the same error. Sorry for all the issues and I appreciate the help. If it is easier to troubleshoot the FTP login errors directly through email, feel free to email me.

Thanks,
Ben

CPKalb Sep 5, 2023
Collaborator

Hi Ben:

I unfortunately don't know much about troubleshooting ftp issues. Let me see if anyone else here is able to help with that.

Christina

jprestop Sep 5, 2023
Maintainer

Hi @balbrightwpc. Please try the ftp server again. I just tried it, entering my full email address as the password and it worked fine:

seneca:jpresto:~> ftp -p ftp.rap.ucar.edu
Connected to franklin.rap.ucar.edu.
220-
220-   ***               National Center for Atmospheric Research              ***
220-   ***                    Research Applications Program                    ***
220-
220-   ***   Use of this system for unauthorized purposes is a violation of    ***
220-   ***   federal law. See Public Law 99-474. Any activity may be logged    ***
220-   ***   and monitored. By using this system, you consent to such logging  ***
220-   ***   and monitoring.                                                   ***
220-
220-   The local time is: Tue Sep 05 12:10:31 2023
220-
220-   If you have problems logging in, please try using a dash (-) as the first
220-   character of your password -- this will turn off the continuation messages
220-   that may be confusing your ftp client.
220-
220-   If you have unusual problems, please contact: [email protected]
220-
220-   ***  NCAR users: *Do not* use your RAL, CIT or UCAS password here!!!    ***
220-   ***  Please contact the RAL Helpdesk if you require assistance using    ***
220-   ***  this FTP service.                                                  ***
220-
220 128.117.14.132 FTP server ready
Name (ftp.rap.ucar.edu:jpresto): anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
230-Login access granted.
230-   Authenticated as user: anonymous
230 Anonymous access granted, restrictions apply
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

jprestop Sep 5, 2023
Maintainer

@balbrightwpc And, when you say "I've also tried adding the (-) as the first character of the password like the instructions say but I got the same error." I'm not sure what you mean. I don't see anything indicating that you should add "the (-) as the first character of the password" in the instructions. Where do you see this?

balbrightwpc Sep 5, 2023
Author

Thank you. I was able to log in this time. Not sure what I did wrong the first time but sorry for the confusion. To answer your question though, in the message after logging into the ftp, it says:

220- If you have problems logging in, please try using a dash (-) as the first
220- character of your password -- this will turn off the continuation messages
220- that may be confusing your ftp client.

So that is what I tried it.

Anyway, I was able to make a directory /incoming/irap/met_help/albright_data and the two netCDF files and config file should be there. If they are not or if you have any problems seeing them, let me know.

Forecast file: ncar_mpas_APCP_ensemble_mean_24HR_2023042400_fhr13-fhr36_INCHES.nc_META.nc
Obs file: urma_2p5km_qpe24_2023042512_in.nc
Config file: ncar_mpas_APCP_ensemble_mean_24HR_2023042400_R1_0.5_in

Thank you,
Ben

CPKalb · 2023-09-05T20:03:47Z

CPKalb
Sep 5, 2023
Collaborator

Hi Ben:

The problem is with your forecast file. It's not in a format that MET can read, although it's close to MET's netCDF format. There are a couple of issues to fix. The first is the global attribute file_type = "NETCDF_NCCF" ;. NETCDF_NCCF specifically refers to CF compliant netCDF files. Since yours is not in this format, it would be good to remove this attribute. Second, the dimensions are inconsistent between the global attributes and the variables. The global attributes list nx = 1799 and nx = 1059. However, at the top of the file, the dimensions say x = 1059 and y = 1799. Additionally, a MET NETCDF file typically uses lat and lon as the dimensions rather than x and y. I ran a quick test removing the file_type global attribute and changing the dimensions to what are listed below:
dimensions:
lon = 1799 ;
lat = 1059 ;
I also added the global attribute hemisphere = "N" ;. With those changes, the file successfully reads into MODE. However, if this change is incorrect (and your grid should be lat = 1799 and lon = 1059), then you would want to instead edit the global attributes so that nx = 1059 and ny = 1799 (rather than changing the dimensions). Either way, you will want to change your dimension names so that x = lon and y = lat.

There is also another caveat. When I read this into MODE, it sets the forecast time to be 19700101_000000 This is the default time MODE uses when it cannot find date/time information in a file. For a MET version netCDF, the time information needs to be stored as attributes of the variable. Your observation file shows how this should be specified. If you want MET to read the date and time information on your forecast data, then you will want to add the attributes below to the apcp_24hr variable:
init_time
init_time_ut
valid_time
valid_time_ut
accum_time
accum_time_sec

Christina

4 replies

balbrightwpc Sep 6, 2023
Author

Thanks so much Christina for all the information. I did not make the forecast file, but will pass the information along to my coworker that did and see if they can make the changes you suggested. I'll let you know if we are successful or not once I get the new file back.

Ben

CPKalb Sep 6, 2023
Collaborator

Hi Ben:

Another option if your coworker is not able to make changes, is to read the data in through python embedding. I'm including the link for a script that sets up a lambert conformal grid in a python embedding script.

Appendix F in the MET document contains the most information about this.

Christina

balbrightwpc Sep 7, 2023
Author

Hi Christina,

We were able to get the file to work with the changes you suggested. Thank you for your help and feel free to mark this thread as closed/completed.

Ben

CPKalb Sep 7, 2023
Collaborator

Hi Ben:

Great. Thanks for marking an answer. I'm going to lock this post. If you have future issues, feel free to start a new discussion.

Christina

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NetCDF File Issue with MODE #2333

{{title}}

Replies: 2 comments 13 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

NetCDF File Issue with MODE #2333

balbrightwpc Aug 31, 2023

Replies: 2 comments · 13 replies

j-opatz Aug 31, 2023 Collaborator

balbrightwpc Sep 5, 2023 Author

CPKalb Sep 5, 2023 Collaborator

jprestop Sep 5, 2023 Maintainer

jprestop Sep 5, 2023 Maintainer

balbrightwpc Sep 5, 2023 Author

CPKalb Sep 5, 2023 Collaborator

balbrightwpc Sep 6, 2023 Author

CPKalb Sep 6, 2023 Collaborator

balbrightwpc Sep 7, 2023 Author

CPKalb Sep 7, 2023 Collaborator

balbrightwpc
Aug 31, 2023

Replies: 2 comments 13 replies

j-opatz
Aug 31, 2023
Collaborator

balbrightwpc Sep 5, 2023
Author

CPKalb Sep 5, 2023
Collaborator

jprestop Sep 5, 2023
Maintainer

jprestop Sep 5, 2023
Maintainer

balbrightwpc Sep 5, 2023
Author

CPKalb
Sep 5, 2023
Collaborator

balbrightwpc Sep 6, 2023
Author

CPKalb Sep 6, 2023
Collaborator

balbrightwpc Sep 7, 2023
Author

CPKalb Sep 7, 2023
Collaborator