enabling szlib compression

[edwardhartnett]

This issue is an offshoot of #23

@junwang-noaa here are instructions for trying szlib compression.

1 - Build HDF5 (1.10.6 for best performance) with szip. Use the --enable-szlib= option to configure. For example:
./configure --with-szlib=/usr/local/szip-2.1.1

At the end of the configure, information will be printed about the build. You should see:
I/O filters (external): deflate(zlib),szip(encoder)

2 - Rebuild netcdf-c with that HDF5 build. NetCDF will detect that szip has been included in HDF5, and you will see this in the information at the end of the configure step:

SZIP Support:           yes
SZIP Write Support:     yes
Parallel Filters:       yes

3 - In your fortran code, do not set the deflate settings, and instead call code like this:

     integer, parameter :: H5_SZIP_NN_OPTION_MASK = 32
     integer, parameter :: H5_SZIP_MAX_PIXELS_PER_BLOCK_IN = 32
     integer, parameter :: HDF5_FILTER_SZIP = 4
C     Set  filter on variable
      params(1) = H5_SZIP_NN_OPTION_MASK
      params(1) = H5_SZIP_MAX_PIXELS_PER_BLOCK_IN
      retval = nf90_def_var_filter(ncid, varid, HDF5_FILTER_SZIP, 1, params)
      if (retval .ne. nf_noerr) stop 1

Just as with nc_def_var_deflate(), this must be called for each variable you want to be compressed.

4 - When done, you can detect filtered data with ncdump -h -s, the variable will have a special attribute like this:
datasetF32:_Filter = "4,169,32,32,2500" ;

[jswhit]

Thanks Ed. Once #23 is merged, I can create a new fork to test this.

[junwang-noaa]

Thanks, Ed and Jeff. We are working on committing #23, will work on the szlib after the commit. Jeff, if you create new fork, please let us know we can test the code on wcoss.

…

On Wed, Jan 29, 2020 at 12:10 PM Jeff Whitaker ***@***.***> wrote: Thanks Ed. Once #23 <#23> is merged, I can create a new fork to test this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AI7D6TKXHGXVTPMD3EEMDV3RAG2A3A5CNFSM4KMZMHLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKH7V4Y#issuecomment-579861235>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TNKQJIZF5PCU5RPZATRAG2A3ANCNFSM4KMZMHLA> .

[jswhit]

is this now in netcdf-c master, or do we need your fork?

[edwardhartnett]

@jswhit everything has been merged to netcdf-c master. No need for any of my branches any longer.

[junwang-noaa]

Great! I am building the netcdf parallel lib under emc.nemspara on hera using the netcdf-c master now.

…

On Wed, Jan 29, 2020 at 2:08 PM Edward Hartnett ***@***.***> wrote: @jswhit <https://github.com/jswhit> everything has been merged to netcdf-c master. No need for any of my branches any longer. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AI7D6TJ42UOK3BZ63ZSNTDTRAHH4BA5CNFSM4KMZMHLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKIL6GQ#issuecomment-579911450>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TMZN76JLKFEY5VDL6LRAHH4BANCNFSM4KMZMHLA> .

[jswhit]

Ed - I went ahead and tried this, and it works! However, with the default parameter settings the file sizes are very large compared to zlib

-rw-r--r-- 1 Jeffrey.S.Whitaker gsienkf  3483360063 Jan 29 19:53 dynf006.nc.zlib
-rw-r--r-- 1 Jeffrey.S.Whitaker gsienkf 17024837145 Jan 29 20:38 dynf006.nc.szip

[edwardhartnett]

But was it faster?

[jswhit]

A little bit (30 secs vs 35 secs).

[edwardhartnett]

OK, good! So it seems to be working and giving faster times, and that's what it's supposed to do. Yes, it does not compress as much. But it's also faster to read. So its a tradeoff.

However, we can also try some other szip settings. Try using:

 integer, parameter :: H5_SZIP_NC_OPTION_MASK = 4

Use this as the szip options mask and see if it is faster or compresses more.

Also try setting H5_SZIP_MAX_PIXELS_PER_BLOCK_IN to 4 and see if that helps.

[jswhit]

Ed - I think the szip filter may not be enabled after all. I don't see the datasetF32:_Filter attribute in the ncdump output, even though I am calling nf90_def_var_filter via

              szip_params(1) = H5_SZIP_NN_OPTION_MASK
              szip_params(2) = H5_SZIP_MAX_PIXELS_PER_BLOCK_IN
              ncerr = nf90_def_var_filter(ncid, varids(i), HDF5_FILTER_SZIP, 2, szip_params)

[jswhit]

OK - I wasn't checking the return code. nf90_def_var_filter is actually failing with

line          224 NetCDF: Invalid argument

I'm using netcdf-fortran 4.5.2 - do I need to update from master?

[edwardhartnett]

Stand by and I will add nf90_def_var_szip to a fortran branch, and you can try that. Probably won't be ready until tomorrow...

[jswhit]

Why doesn't nf90_def_var_filter work as in your example above?

[edwardhartnett]

I don't know. ;-) But I have to add nf90_def_var_szip to fortran, and I will add a test, and then I will know that it works. ;-)

[edwardhartnett]

OK, all this has been added to netcdf-c, do I will close this issue.

Unfortunately, szip with parallel writes is not working. ;-( I have raised an issue with the HDF5 team but it's unlikely this will get much attention in the short term.

The good news is that Charlie Zender and I are soon coming out with a new release of the CCR package, which will add more compression options to netCDF. ;-)

The netcdf-c filter behavior of 4.7.2, 4.7.3, and 4.7.4 has been a bit rocky, which explains some of the confusion about this. There have been changes in approach at Unidata and we have settled on an API and behavior which is guaranteed to be stable moving forward, starting in release 4.8.0.