cfgrib misses some variables in GFS grib data

[meteoDaniel]

Dear all,
I am working with some historical GFS grib files. Not all Variables that are in the file (checked with wgrib2 -v) could be read.

1:0:d=2017020212:TMP Temperature [K]:850 mb:324 hour fcst:
2:2188:d=2017020212:RH Relative Humidity [%]:850 mb:324 hour fcst:
3:4879:d=2017020212:TMP Temperature [K]:1000 mb:324 hour fcst:
4:7067:d=2017020212:RH Relative Humidity [%]:1000 mb:324 hour fcst:
5:9507:d=2017020212:PRES Pressure [Pa]:surface:324 hour fcst:
6:13453:d=2017020212:TMP Temperature [K]:surface:324 hour fcst:
7:15893:d=2017020212:SNOD Snow Depth [m]:surface:324 hour fcst:
8:17771:d=2017020212:TMP Temperature [K]:2 m above ground:324 hour fcst:
9:20964:d=2017020212:DPT Dew Point Temperature [K]:2 m above ground:324 hour fcst:
10:23404:d=2017020212:RH Relative Humidity [%]:2 m above ground:324 hour fcst:
11:25592:d=2017020212:SUNSD Sunshine Duration [s]:surface:324 hour fcst:
12:29287:d=2017020212:DSWRF Downward Short-Wave Radiation Flux [W/m^2]:surface:312-324 hour ave fcst:
13:31248:d=2017020212:DLWRF Downward Long-Wave Rad. Flux [W/m^2]:surface:312-324 hour ave fcst:
14:33209:d=2017020212:USWRF Upward Short-Wave Radiation Flux [W/m^2]:surface:312-324 hour ave fcst:
15:34919:d=2017020212:ULWRF Upward Long-Wave Rad. Flux [W/m^2]:surface:312-324 hour ave fcst:

Only the

SNOD
SUNSD
RH
TMP
PRES

variables and one of the radiation could be detected by cfgrib xarray_store.open_dataset()

gfs.0p25.2017020212.f324.grib2.spasub.zip

[alexamici]

Thanks for the report and for the file sample.

I confirm the silent dropping of variables, and at first look, the file appear to trigger two independent bugs.

To recover the three heightAboveGround variables you can pass filter_by_keys={'typeOfLevel': 'heightAboveGround'}. Not sure why it is not in the list of filters to try.

[meteoDaniel]

Thanks @alexamici

[alexamici]

The first half of the issue is solved in bf3c56a and, I must say, was a really nice bug. It made cfgrib.xarray_store.open_datasets missing a ton of variables in the nam example GRIB.

There are four more variable missing, the one with a step interval:

edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType  
2            kwbc         20170202     fc           regular_ll   312-324      surface      0            dswrf        grid_simple 
2            kwbc         20170202     fc           regular_ll   312-324      surface      0            dlwrf        grid_simple 
2            kwbc         20170202     fc           regular_ll   312-324      surface      0            uswrf        grid_simple 
2            kwbc         20170202     fc           regular_ll   312-324      surface      0            ulwrf        grid_simple 
1

[alexamici]

@meteoDaniel both issues should be fixed in master, please let me know is it works for you by installing temporarily with:

$ pip install git+https://github.com/ecmwf/cfgrib

That single file uncovered not one, but three subtle and damaging bugs, thanks!

[meteoDaniel]

Works fine for me ! Great support. I like cfgrib and recommend it to all i am talking about grib data.