netcdf built with hdf5 1.10.8 error after 'nc_def_var_fletcher32'

[abhibaruah]

NetCDF version: v4.8.1
OS: Linux 10
Compiler gcc 8.3.0

I am a developer at The MathWorks Inc. For MATLAB, we were building netcdf with hdf5-1.8.12.
However, we are trying to change netcdf's dependency on hdf5 to v1.10.8.

I built netcdf successfully with hdf5 1.10.8. However, while qualifying I started seeing some of our tests fail, which I have narrowed down to an issue with 'nc_def_var_fletcher'.

I am attaching a C repro code below, which works fine when netcdf 4.8.1 is built with hdf5 1.8.12.

However, when netcdf 4.8.1 is built with hdf5 1.10.8, 'nc_close' on line 35 gives the error code -101 (NC_EHDFERR)
The error does not occur if the datatype (xtype) in 'nc_def_var' is anything other than NC_STRING.
Also, the error does not occur if the call to 'nc_def_var_fletcher32' is removed.

Any help in resolving this would be highly appreciated.

#include <iostream>
#include <vector>
#include "netcdf.h"

void checkErrorCode(int status, const char* message){
    if (status != NC_NOERR){
        std::cout << "Error code: " << status << " from " << message << std::endl;
    }
}

int main(int argc, const char * argv[]) {
    
    int ncid;
    int retval;
    
    retval = nc_create("myfile.nc", NC_NETCDF4, &ncid);
    checkErrorCode(retval, "nc_create");
    
    // Define dimensions
    int dimid_a;
	int dimlen = 100;
    retval = nc_def_dim(ncid, "a", dimlen, &dimid_a);
    checkErrorCode(retval, "nc_def_dim for 'a'");
    
    // Define variable
    int varid;
	int dimids[1] = {dimid_a};
    retval = nc_def_var(ncid, "var", NC_STRING, 1, dimids, &varid);
    checkErrorCode(retval, "nc_def_var");
        
    
    retval = nc_def_var_fletcher32(ncid, varid, true);
    checkErrorCode(retval, "nc_def_var_fletcher32");

    retval = nc_close(ncid);
    checkErrorCode(retval, "nc_close");
    
    
    return retval;
}

[WardF]

Thanks; taking a look at it now, I will also see about getting 1.10.8 into our test matrix.

[WardF]

What platform are you observing this failure on?

[WardF]

Nevermind my question, I see it in the header of the issue you opened, sorry, thanks :)

[WardF]

Observing this issue on OSX as well.

[WardF]

In linux with debug and logging turned on, observing the following.

TLDR: H5Dcreate2 is throwing an error for some reason.

wfisher@ubuntudev-arm:~/Desktop/github-test$ ./test 
				log_level changed to 5
		HDF5_def_dim: ncid 0x10000 name a len 100
				nc4_find_dim: dimid 0
		NC4_def_var: name var type 12 ndims 1
				dimid[0] 0
				nc4_get_typelen_mem xtype: 12
		nc_inq_atomic_type: typeid 12
				nc4_type_new: size 8 name string assignedid 12
				nc4_find_dim: dimid 0
				allocating array of 1 size_t to hold chunksizes for var var
				nc4_find_default_chunksizes2: name var dim 0 DEFAULT_CHUNK_SIZE 4194304 num_values 100.000000 type_size 8 chunksize 100
				total_chunk_size 800.000000
				nc4_get_typelen_mem xtype: 12
				new varid 0
		nc_def_var_extra: ncid 0x10000 varid 0
			extra: default: chunksizes for var varsizes=100
	NC4_close: ncid 0x10000
			nc4_close_hdf5_file: h5->path myfile.nc abort 0
			sync_netcdf4_file
		*** NetCDF-4 Internal Metadata: int_ncid 0x0 ext_ncid 0x10000
		FILE - path: myfile.nc cmode: 0x11009 parallel: 0 redef: 0 fill_mode: 0 no_write: 0 next_nc_grpid: 1
		 GROUP - / nc_grpid: 0 nvars: 1 natts: 0
		 DIMENSION - dimid: 0 name: a len: 100 unlimited: 0
		 VARIABLE - varid: 0 name: var ndims: 1 dimids: 0 storage: chunked
			nc4_rec_write_groups_types: grp->hdr.name /
			nc4_rec_write_metadata: grp->hdr.name /, bad_coord_order 0
				nc4_create_dim_wo_var: creating dim a
				nc4_create_dim_wo_var: about to H5Dcreate1 a dimscale dataset a
				write_netcdf4_dimid: writing secret dimid 0
				write_var: writing var var
			var_create_dataset:: name var
				var_create_dataset: about to H5Dcreate2 dataset var of type 0x13e
ERROR: file nc4hdf.c, line 988.
NetCDF: HDF error
Error -101: nc_close was received

[WardF]

For comparison, the following succeeds (netCDF 4.8.1, HDF5 1.8.21)

wfisher@ubuntudev-arm:~/Desktop/github-test$ ./test 
				log_level changed to 5
		HDF5_def_dim: ncid 0x10000 name a len 100
				nc4_find_dim: dimid 0
		NC4_def_var: name var type 12 ndims 1
				dimid[0] 0
				nc4_get_typelen_mem xtype: 12
		nc_inq_atomic_type: typeid 12
				nc4_type_new: size 8 name string assignedid 12
				nc4_find_dim: dimid 0
				allocating array of 1 size_t to hold chunksizes for var var
				nc4_find_default_chunksizes2: name var dim 0 DEFAULT_CHUNK_SIZE 4194304 num_values 100.000000 type_size 8 chunksize 100
				total_chunk_size 800.000000
				nc4_get_typelen_mem xtype: 12
				new varid 0
		nc_def_var_extra: ncid 0x10000 varid 0
			extra: default: chunksizes for var varsizes=100
	NC4_close: ncid 0x10000
			nc4_close_hdf5_file: h5->path myfile.nc abort 0
			sync_netcdf4_file
		*** NetCDF-4 Internal Metadata: int_ncid 0x0 ext_ncid 0x10000
		FILE - path: myfile.nc cmode: 0x11009 parallel: 0 redef: 0 fill_mode: 0 no_write: 0 next_nc_grpid: 1
		 GROUP - / nc_grpid: 0 nvars: 1 natts: 0
		 DIMENSION - dimid: 0 name: a len: 100 unlimited: 0
		 VARIABLE - varid: 0 name: var ndims: 1 dimids: 0 storage: chunked
			nc4_rec_write_groups_types: grp->hdr.name /
			nc4_rec_write_metadata: grp->hdr.name /, bad_coord_order 0
				nc4_create_dim_wo_var: creating dim a
				nc4_create_dim_wo_var: about to H5Dcreate1 a dimscale dataset a
				write_netcdf4_dimid: writing secret dimid 0
				write_var: writing var var
			var_create_dataset:: name var
				var_create_dataset: about to H5Dcreate2 dataset var of type 0x3000060
		attach_dimscales: attaching scale for dimid 0 to var var
					NC4_write_ncproperties
			nc4_rec_grp_HDF5_del: grp->name /
			closing HDF5 dataset 83886081
				nc4_rec_grp_HDF5_del: closing group /
			nc4_close_netcdf4_file: h5->path myfile.nc abort 0
					NC4_clear_provenance
			nc4_rec_grp_del: grp->name /
				var_free: deleting var var
				nc4_type_free: deleting type string
				dim_free: deleting dim a

[WardF]

Note to self: Potential resource to follow up on:

• https://forum.hdfgroup.org/t/fletcher32-filter-on-variable-length-string-datasets-not-suitable-for-filters/9038

[WardF]

Note to self, I've narrowed this down as a regression between HDF5 1.10.3 (works) and 1.10.7 (fails). It is unclear to me as of yet whether or not fletcher32 on a variable-length datatype variable is valid in the HDF5 API. I'll investigate this further tomorrow.

[WardF]

Confirmation that HDF5 1.10.6 appears to work, I'll have to dig into the release notes.

[WardF]

From the 1.10.7 release notes, I see the following note. It is confusing since it seems to indicate, at a glance, the reverse of what we are observing. It will merit further review.

   - Creation of dataset with optional filter

      When the combination of type, space, etc doesn't work for filter
      and the filter is optional, it was supposed to be skipped but it was
      not skipped and the creation failed.

      A fix is applied to allow the creation of a dataset in such
      situation, as specified in the user documentation.

      (BMR - 2020/8/13, HDFFV-10933)

[WardF]

I'm still taking a look into why this is happening, but the bottom line might be that it might be beyond our control in this circumstance. It should not, however, fail silently. We'll have to put some thought into how we will best handle this at the netcdf layer.

[DennisHeimbigner]

We could put in a check in nc_def_var_filter so that if one attempts to add a filter to
a string or vlen typed variable, the call to nc_def_var_filter will fail.

[WardF]

@DennisHeimbigner Agreed. I'm uncertain (and can't switch gears at the moment to think about it) whether we want to remove the filter definition when the combination doesn't work and silently proceed, or whether we want to fail with an informative message. The question is whether we want to break a workflow with a message that explains why, or do we want to simply retain previous behavior (one layer up).

[WardF]

@abhibaruah for what it's worth, it looks like this test may need some alteration on your end, regardless of how we proceed. Any HDF5 version 1.10.7+ is going to (from my testing) result in the behavior you're seeing.

[DennisHeimbigner]

I interpret the HDF5 release notes to mean that if the filter cannot be applied, then it is now
skipped, whereas before it was not and was executed anyway. My interpretations is that this
relates to a function
in the filter API called "can_apply". It is invoked before a filter is used to indicate that the filter
should work or not on the specific variable. So, for fletcher32, this function probably is defined to
indicate that it cannot apply if given a variable of variable length type. However, the HDF5 library
was ignoring this and going ahead and applying it anyway. The fix in 1.10.7 probably causes
the library to pay attention to the output of can_apply(). Not sure when can_apply() is called.
Does that interpretation fit the symptoms?

[abhibaruah]

Hello Ward and Dennis,
Thank you for working on this.
My understanding from the discussion above is that from hdf5 1.10.7 onwards, invoking a filter on a variable of variable length type will fail.
However, this behavior seems to exist with only fletcher32. I tried using the netcdf function 'nc_def_var_deflate' on a string type variable, and it did not throw any error (I am unsure if the filter is applied or if it is simply ignored).

I wonder if the erroring behavior with fletcher32 might have been an oversight on the part of hdf5.

[abhibaruah]

Meanwhile, I have also reached out to the HDF group and forwarded them the link to this discussion. Hopefully, they can help us in identifying next steps for the same, and if the behavior in the hdf5 layer is intended or a bug.

[DennisHeimbigner]

My suspicion is that other filters do not have a can_apply() function or the function
does not forbid variable length types. If correct, then what is compressed is not
the actual data but rather the pointers to the variable length data; the pointer to a string
or the .p field of an hvl_t object.

[abhibaruah]

You are correct Dennis.
Here is what Elena from THG replied:
"Yes, behavior changed. Filtering for variable-size types was disabled at some point by Quincey."

She also confirmed that the erroring behavior would be seen only with fletcher32, since the checksum is mandatory. For all other filters, they will be silently ignored when used with variables having variable length datatypes.

Here is what Elena wrote:
"See H5ocpl.c file where gzip and fletcher filtesr are registered (i.e., H5Pset_deflate and H5Pset_fletcher32). The first filter is optional while checksum is mandatory."

[abhibaruah]

@DennisHeimbigner Agreed. I'm uncertain (and can't switch gears at the moment to think about it) whether we want to remove the filter definition when the combination doesn't work and silently proceed, or whether we want to fail with an informative message. The question is whether we want to break a workflow with a message that explains why, or do we want to simply retain previous behavior (one layer up).

So, with reference to the two options in Ward's previous comment, which option will the netcdf library go ahead with? Ignoring the filter and proceeding or failing with a more informative message?

[DennisHeimbigner]

Frankly, I have no idea how to disentangle this.
Needs some thought.

[DennisHeimbigner]

After some discussion, we have decided to treat this an an error.

[DennisHeimbigner]

"Fixed" by #2231