Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zarr: missing fill_value on creation #2059

Closed
rouault opened this issue Aug 9, 2021 · 0 comments · Fixed by #2066
Closed

Zarr: missing fill_value on creation #2059

rouault opened this issue Aug 9, 2021 · 0 comments · Fixed by #2066

Comments

@rouault
Copy link
Contributor

rouault commented Aug 9, 2021

netcdf-c master at 84f0696 , on Ubuntu 20.04

given groups.h5,
./ncdump/nccopy groups.h5 "file://groups_regular.zarr#mode=zarr,file"

generates groups_regular.zarr/MyGroup/dset1/.zarray with

{"zarr_format": 2,"shape": [3,3],"dtype": ">i4","chunks": [3,3],"order": "C","compressor": null,"filters": null}

There's no fill_value key, which is required by the Zarr specification (https://zarr.readthedocs.io/en/stable/spec/v2.html#metadata), and should be at null

This causes the Zarr python implementation to reject it (as well as the new GDAL Zarr driver: https://gdal.org/drivers/raster/zarr.html):

$ python
Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zarr
>>> z = zarr.open('/home/even/netcdf-c/build/groups.zarr')
>>> z['/MyGroup/dset1']
Traceback (most recent call last):
  File "/home/even/.local/lib/python3.8/site-packages/zarr/meta.py", line 43, in decode_array_metadata
    fill_value = decode_fill_value(meta['fill_value'], dtype)
KeyError: 'fill_value'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/even/.local/lib/python3.8/site-packages/zarr/hierarchy.py", line 341, in __getitem__
    return Array(self._store, read_only=self._read_only, path=path,
  File "/home/even/.local/lib/python3.8/site-packages/zarr/core.py", line 159, in __init__
    self._load_metadata()
  File "/home/even/.local/lib/python3.8/site-packages/zarr/core.py", line 176, in _load_metadata
    self._load_metadata_nosync()
  File "/home/even/.local/lib/python3.8/site-packages/zarr/core.py", line 191, in _load_metadata_nosync
    meta = decode_array_metadata(meta_bytes)
  File "/home/even/.local/lib/python3.8/site-packages/zarr/meta.py", line 57, in decode_array_metadata
    raise MetadataError('error decoding metadata: %s' % e)
zarr.errors.MetadataError: error decoding metadata: 'fill_value'
>>> 
DennisHeimbigner added a commit to DennisHeimbigner/netcdf-c that referenced this issue Aug 9, 2021
re: Issues Unidata#2063, Unidata#2062, Unidata#2061, Unidata#2059

1. Support "fill_value: null" (Unidata#2063).
2. Handle the dtype case "|u1" (Unidata#2062).
3. When writing a pure Zarr format file, some nczarr attributes inadvertently crept in (Unidata#2061).
4. If there is no fill value, then the .zarray fill_value key should have the value null rather than left out (Unidata#2059).

Hat tip: Even Rouault
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant