Set _FillValue

[martindurant]

Fixes #177

[rsignell-usgs]

@martindurant , I thought we wanted to set fill_value to be the value of _FillValue, not vice versa:
#177 (comment)

[martindurant]

The code linked in the issue is what I was going on

[martindurant]

@rabernat to choose: when creating the zarr output metadata, fill_value overwrites _FillValue, or the reverse?

[keewis]

as far as I understand #177 (comment), we should always ignore the HDF5 dataset fill value (fill_value) because it is static and not part of the netcdf4 spec.

[martindurant]

@keewis , so we should only use it when _FillValue is not available? What about if the dataset is not netCDF, but regular HDF?

[keewis]

according to https://docs.unidata.ucar.edu/netcdf-c/current/attribute_conventions.html:

It is not necessary to define your own _FillValue attribute for a variable if the default fill value for the type of the variable is adequate. However, use of the default fill value for data type byte is not recommended.

So that means you'd use the default fill value defined by the netcdf4 library if _FillValue is not set?

For HDF5 datasets it seems the correct thing to do is what kerchunk currently does: translate the HDF5 fill_value property to zarr's fill_value (and you will get a default if that is not set).

---

Re-reading the comments in #177, I get the impression that @ajelenak and @rabernat actually disagree on what to do (I might very well be misunderstanding something, though): @ajelenak would prefer to expose both fill_value and _FillValue from xarray's zarr backend and have decode_cf decide which to use, while @rabernat would like to have kerchunk set zarr's fill_value to _FillValue for netcdf4 datasets. Maybe they can clarify?

[rabernat]

It's not a question of what I would like or not like. My goal was to communicate how Xarray handles fill_value with Zarr.

1. When opening a Zarr store for reading, the Zarr array .fill_value is copied into the CF-style attribute _FillValue. So when the data are decoded whatever the the fill_value is presented to the user as NaN. In the code, you can see that any existing _FillValue attribute is overwritten. That is potentially a problem. However, that situation would never arise if you are using Xarray to write the data because...
2. When writing an array, the _FillValue attribute is removed from attributes and placed in fill_value of the zarr array (code).

Creating a kerchunk Zarr is analogous to Xarray writing a Zarr store. So I recommend you follow the same pattern here. If _FillValue is present in the attributes, move it .fill_value of the array. Do not set the _FillValue attribute. Xarray will set it when decoding the data.

[rsignell-usgs]

@martindurant, as I understand it, the new plan is to modify the PR so that in the Kerchunk JSON we:

• assign the value of _FillValue to fill_value.
• remove the _FillValue attribute (to avoid confusion)

Correct?

[keewis]

to follow @rabernat's recommendation kerchunk and still keep supporting HDF datasets, I think we need to do a slight modification of that:

• detect the dataset type (netcdf or HDF)
• get the appropriate fill value (_FillValue for netcdf or the dataset fill value for HDF)
• assign that to zarr's fill_value
• clear any _FillValue attributes

In any case, @martindurant: I think this PR would fix #105 as well

[martindurant]

detect the dataset type (netcdf or HDF)

I'm not sure this is a thing: all netCDF4 files are valid HDF5, but not vice versa; plus you can reasonably store netCDF datasets as parts of an HDF5 at some subpath.

[keewis]

well, the point is that netcdf4 does not use or set the HDF5 fill value, as far as I can tell, so we need to figure out which fill value to use with a particular dataset. That "detection" does not have to happen in code, though, it could just as well be a parameter to SingleHdfToZarr.

[rsignell-usgs]

@keewis I see your point -- if the HDF5 file is generated with NetCDF4, it will have _FillValue defined and we should use that for the Kerchunk fill_value, but if not we should use the HDF5 fill_value for the Kerchunk fill_value.

Is there a fast/definitive way to tell if an HDF5 file was generated by NetCDF4?

[martindurant]

How about this version, then? Would someone like to test?

[rsignell-usgs]

Working! https://nbviewer.org/gist/ae2dc327960c41e97a97e512bdc55a30

[keewis]

Is there a fast/definitive way to tell if an HDF5 file was generated by NetCDF4?

It is probably not definitive, but https://github.com/pedro-vicente/netcdf-detect uses some internal attributes that the netcdf library sets to detect that. It is written in C++, but it should be possible to do the same using h5py (see also this post and the follow-ups on the netcdf mailing list).

[rsignell-usgs]

@keewis , so basically, check for:

1. an attribute named "_Netcdf4Dimid" (in some cases)
2. an attribute named "NAME", (always), saved by the HDF5 Dimension Scales API, that contains the string "This is a netCDF > dimension but not a netCDF variable."

This utility tries to detect both attributes by traversing the HDF5 file, if either case is found, it returns a value of 1