RuntimeError: NetCDF: Filter error: bad id or parameters or duplicate filter

[kfarzad]

Hi, I am trying the MELODIES-MONET/examples/process_obs/reformat_airnow.ipynb, and in the last cell, I am facing the following issue:
Could you please help me with it?

#wite out to filename set in cell 2
write_util.write_ncf(t,filename)

Writing: AIRNOW_20190901_20190930.nc
Compressing: BARPR, original_dtype: float64
Compressing: BC, original_dtype: float64
Compressing: CO, original_dtype: float64
Compressing: NO, original_dtype: float64
Compressing: NO2, original_dtype: float64
Compressing: NO2Y, original_dtype: float64
Compressing: NOX, original_dtype: float64
Compressing: NOY, original_dtype: float64
Compressing: OZONE, original_dtype: float64
Compressing: PM10, original_dtype: float64
Compressing: PM2.5, original_dtype: float64
Compressing: PRECIP, original_dtype: float64
Compressing: RHUM, original_dtype: float64
Compressing: RWD, original_dtype: float64
Compressing: RWS, original_dtype: float64
Compressing: SO2, original_dtype: float64
Compressing: SRAD, original_dtype: float64
Compressing: TEMP, original_dtype: float64
Compressing: UV-AETH, original_dtype: float64
Compressing: WD, original_dtype: float64
Compressing: WS, original_dtype: float64
Compressing: cmsa_name, original_dtype: float64
c:\Users\farzad.k\miniconda3\envs\melodies-monet\lib\site-packages\xarray\core\duck_array_ops.py:187: RuntimeWarning: invalid value encountered in cast
return data.astype(dtype, **kwargs)
Compressing: msa_code, original_dtype: float64

RuntimeError Traceback (most recent call last)
Cell In[7], line 2
1 #wite out to filename set in cell 2
----> 2 write_util.write_ncf(t,filename)

File c:\Users\farzad.k\miniconda3\envs\melodies-monet\lib\site-packages\melodies_monet\util\write_util.py:37, in write_ncf(dset, output_name, title)
35 dset.attrs['format'] = 'NetCDF-4'
36 dset.attrs['date_created'] = pd.to_datetime('today').strftime('%Y-%m-%d')
---> 37 dset.to_netcdf(output_name, encoding=encoding)

File c:\Users\farzad.k\miniconda3\envs\melodies-monet\lib\site-packages\xarray\core\dataset.py:1911, in Dataset.to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
1908 encoding = {}
1909 from xarray.backends.api import to_netcdf
-> 1911 return to_netcdf( # type: ignore # mypy cannot resolve the overloads:(
1912 self,
1913 path,
1914 mode=mode,
1915 format=format,
1916 group=group,
1917 engine=engine,
1918 encoding=encoding,
1919 unlimited_dims=unlimited_dims,
1920 compute=compute,
1921 multifile=False,
1922 invalid_netcdf=invalid_netcdf,
1923 )

File c:\Users\farzad.k\miniconda3\envs\melodies-monet\lib\site-packages\xarray\backends\api.py:1217, in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
1212 # TODO: figure out how to refactor this logic (here and in save_mfdataset)
1213 # to avoid this mess of conditionals
1214 try:
1215 # TODO: allow this work (setting up the file for writing array data)
1216 # to be parallelized with dask
-> 1217 dump_to_store(
1218 dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
1219 )
1220 if autoclose:
1221 store.close()

File c:\Users\farzad.k\miniconda3\envs\melodies-monet\lib\site-packages\xarray\backends\api.py:1264, in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
1261 if encoder:
1262 variables, attrs = encoder(variables, attrs)
-> 1264 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

File c:\Users\farzad.k\miniconda3\envs\melodies-monet\lib\site-packages\xarray\backends\common.py:271, in AbstractWritableDataStore.store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
269 self.set_attributes(attributes)
270 self.set_dimensions(variables, unlimited_dims=unlimited_dims)
--> 271 self.set_variables(
272 variables, check_encoding_set, writer, unlimited_dims=unlimited_dims
273 )

File c:\Users\farzad.k\miniconda3\envs\melodies-monet\lib\site-packages\xarray\backends\common.py:309, in AbstractWritableDataStore.set_variables(self, variables, check_encoding_set, writer, unlimited_dims)
307 name = _encode_variable_name(vn)
308 check = vn in check_encoding_set
--> 309 target, source = self.prepare_variable(
310 name, v, check, unlimited_dims=unlimited_dims
311 )
313 writer.add(source, target)

File c:\Users\farzad.k\miniconda3\envs\melodies-monet\lib\site-packages\xarray\backends\netCDF4_.py:488, in NetCDF4DataStore.prepare_variable(self, name, variable, check_encoding, unlimited_dims)
486 nc4_var = self.ds.variables[name]
487 else:
--> 488 nc4_var = self.ds.createVariable(
489 varname=name,
490 datatype=datatype,
491 dimensions=variable.dims,
492 zlib=encoding.get("zlib", False),
493 complevel=encoding.get("complevel", 4),
494 shuffle=encoding.get("shuffle", True),
495 fletcher32=encoding.get("fletcher32", False),
496 contiguous=encoding.get("contiguous", False),
497 chunksizes=encoding.get("chunksizes"),
498 endian="native",
499 least_significant_digit=encoding.get("least_significant_digit"),
500 fill_value=fill_value,
501 )
503 nc4_var.setncatts(attrs)
505 target = NetCDF4ArrayWrapper(name, self)

File src\netCDF4_netCDF4.pyx:2945, in netCDF4._netCDF4.Dataset.createVariable()

File src\netCDF4_netCDF4.pyx:4184, in netCDF4._netCDF4.Variable.init()

File src\netCDF4_netCDF4.pyx:2014, in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Filter error: bad id or parameters or duplicate filter

[zmoon]

Hi @kiarashFarzad , thanks for reporting this. We recently addressed this in our develop branch, see #162 .

[zmoon]

The example should work if you use our develop branch or downgrade to 'libnetcdf<4.9' in your environment.

[kfarzad]

Hi @kiarashFarzad , thanks for reporting this. We recently addressed this in our develop branch, see #162 .

Thanks a lot!