v2.0.rst

.. _spec_v2:

Zarr Storage Specification Version 2
====================================

This document provides a technical specification of the protocol and format
used for storing Zarr arrays. The key words "MUST", "MUST NOT", "REQUIRED",
"SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in `RFC 2119
<https://www.ietf.org/rfc/rfc2119.txt>`_.

Status
------

This specification is the latest version. See :ref:`spec` for previous
versions.

.. _spec_v2_storage:

Storage
-------

A Zarr array can be stored in any storage system that provides a key/value
interface, where a key is an ASCII string and a value is an arbitrary sequence
of bytes, and the supported operations are read (get the sequence of bytes
associated with a given key), write (set the sequence of bytes associated with
a given key) and delete (remove a key/value pair).

For example, a directory in a file system can provide this interface, where
keys are file names, values are file contents, and files can be read, written
or deleted via the operating system. Equally, an S3 bucket can provide this
interface, where keys are resource names, values are resource contents, and
resources can be read, written or deleted via HTTP.

Below an "array store" refers to any system implementing this interface.

.. _spec_v2_array:

Arrays
------

.. _spec_v2_array_metadata:

Metadata
~~~~~~~~

Each array requires essential configuration metadata to be stored, enabling
correct interpretation of the stored data. This metadata is encoded using JSON
and stored as the value of the ".zarray" key within an array store.

The metadata resource is a JSON object. The following keys MUST be present
within the object:

zarr_format
    An integer defining the version of the storage specification to which the
    array store adheres.
shape
    A list of integers defining the length of each dimension of the array.
chunks
    A list of integers defining the length of each dimension of a chunk of the
    array. Note that all chunks within a Zarr array have the same shape.
dtype
    A string or list defining a valid data type for the array. See also
    the subsection below on data type encoding.
compressor
    A JSON object identifying the primary compression codec and providing
    configuration parameters, or ``null`` if no compressor is to be used.
    The object MUST contain an ``"id"`` key identifying the codec to be used.
fill_value
    A scalar value providing the default value to use for uninitialized
    portions of the array, or ``null`` if no fill_value is to be used.
order
    Either "C" or "F", defining the layout of bytes within each chunk of the
    array. "C" means row-major order, i.e., the last dimension varies fastest;
    "F" means column-major order, i.e., the first dimension varies fastest.
filters
    A list of JSON objects providing codec configurations, or ``null`` if no
    filters are to be applied. Each codec configuration object MUST contain a
    ``"id"`` key identifying the codec to be used.

The following keys MAY be present within the object:

dimension_separator
    If present, either the string ``"."`` or ``"/"`` defining the separator placed
    between the dimensions of a chunk. If the value is not set, then the
    default MUST be assumed to be ``"."``, leading to chunk keys of the form "0.0".
    Arrays defined with ``"/"`` as the dimension separator can be considered to have
    nested, or hierarchical, keys of the form "0/0" that SHOULD where possible
    produce a directory-like structure.

Other keys SHOULD NOT be present within the metadata object and SHOULD be
ignored by implementations.

For example, the JSON object below defines a 2-dimensional array of 64-bit
little-endian floating point numbers with 10000 rows and 10000 columns, divided
into chunks of 1000 rows and 1000 columns (so there will be 100 chunks in total
arranged in a 10 by 10 grid). Within each chunk the data are laid out in C
contiguous order. Each chunk is encoded using a delta filter and compressed
using the Blosc compression library prior to storage::

    {
        "chunks": [
            1000,
            1000
        ],
        "compressor": {
            "id": "blosc",
            "cname": "lz4",
            "clevel": 5,
            "shuffle": 1
        },
        "dtype": "<f8",
        "fill_value": "NaN",
        "filters": [
            {"id": "delta", "dtype": "<f8", "astype": "<f4"}
        ],
        "order": "C",
        "shape": [
            10000,
            10000
        ],
        "zarr_format": 2
    }

.. _spec_v2_array_dtype:

Data type encoding
~~~~~~~~~~~~~~~~~~

Simple data types are encoded within the array metadata as a string,
following the `NumPy array protocol type string (typestr) format <https://numpy.org/doc/stable/reference/arrays.interface.html#the-array-interface-protocol>`_. The format
consists of 3 parts:

* One character describing the byteorder of the data (``"<"``: little-endian;
  ``">"``: big-endian; ``"|"``: not-relevant)
* One character code giving the basic type of the array (``"b"``: Boolean (integer
  type where all values are only True or False); ``"i"``: integer; ``"u"``: unsigned
  integer; ``"f"``: floating point; ``"c"``: complex floating point; ``"m"``: timedelta;
  ``"M"``: datetime; ``"S"``: string (fixed-length sequence of char); ``"U"``: unicode
  (fixed-length sequence of Py_UNICODE); ``"V"``: other (void * – each item is a
  fixed-size chunk of memory))
* An integer specifying the number of bytes the type uses.

The byte order MUST be specified. E.g., ``"<f8"``, ``">i4"``, ``"|b1"`` and
``"|S12"`` are valid data type encodings.

For datetime64 ("M") and timedelta64 ("m") data types, these MUST also include the
units within square brackets. A list of valid units and their definitions are given in
the `NumPy documentation on Datetimes and Timedeltas <https://numpy.org/doc/stable/reference/arrays.datetime.html#datetime-units>`_.
For example, ``"<M8[ns]"`` specifies a datetime64 data type with nanosecond time units.

Structured data types (i.e., with multiple named fields) are encoded
as a list of lists, following `NumPy array protocol type descriptions
(descr) <https://numpy.org/doc/stable/reference/arrays.interface.html#the-array-interface-protocol>`_. Each
sub-list has the form ``[fieldname, datatype, shape]`` where ``shape``
is optional. ``fieldname`` is a string, ``datatype`` is a string
specifying a simple data type (see above), and ``shape`` is a list of
integers specifying subarray shape. For example, the JSON list below
defines a data type composed of three single-byte unsigned integer
fields named "r", "g" and "b"::

    [["r", "|u1"], ["g", "|u1"], ["b", "|u1"]]

For example, the JSON list below defines a data type composed of three
fields named "x", "y" and "z", where "x" and "y" each contain 32-bit
floats, and each item in "z" is a 2 by 2 array of floats::

    [["x", "<f4"], ["y", "<f4"], ["z", "<f4", [2, 2]]]

Structured data types may also be nested, e.g., the following JSON
list defines a data type with two fields "foo" and "bar", where "bar"
has two sub-fields "baz" and "qux"::

    [["foo", "<f4"], ["bar", [["baz", "<f4"], ["qux", "<i4"]]]]

.. _spec_v2_array_fill_value:

Fill value encoding
~~~~~~~~~~~~~~~~~~~

For simple floating point data types, the following table MUST be used to
encode values of the "fill_value" field:

=================  ===============
Value              JSON encoding
=================  ===============
Not a Number       ``"NaN"``
Positive Infinity  ``"Infinity"``
Negative Infinity  ``"-Infinity"``
=================  ===============

If an array has a fixed length byte string data type (e.g., ``"|S12"``), or a
structured data type, and if the fill value is not null, then the fill value
MUST be encoded as an ASCII string using the standard Base64 alphabet.

.. _spec_v2_array_chunks:

Chunks
~~~~~~

Each chunk of the array is compressed by passing the raw bytes for the chunk
through the primary compression library to obtain a new sequence of bytes
comprising the compressed chunk data. No header is added to the compressed
bytes or any other modification made. The internal structure of the compressed
bytes will depend on which primary compressor was used. For example, the `Blosc
compressor <https://github.com/Blosc/c-blosc/blob/main/README_CHUNK_FORMAT.rst>`_
produces a sequence of bytes that begins with a 16-byte header followed by
compressed data.

The compressed sequence of bytes for each chunk is stored under a key formed
from the index of the chunk within the grid of chunks representing the array.
To form a string key for a chunk, the indices are converted to strings and
concatenated with the period character (".") separating each index. For
example, given an array with shape (10000, 10000) and chunk shape (1000, 1000)
there will be 100 chunks laid out in a 10 by 10 grid. The chunk with indices
(0, 0) provides data for rows 0-999 and columns 0-999 and is stored under the
key "0.0"; the chunk with indices (2, 4) provides data for rows 2000-2999 and
columns 4000-4999 and is stored under the key "2.4"; etc.

There is no need for all chunks to be present within an array store. If a chunk
is not present then it is considered to be in an uninitialized state.  An
uninitialized chunk MUST be treated as if it was uniformly filled with the value
of the "fill_value" field in the array metadata. If the "fill_value" field is
``null`` then the contents of the chunk are undefined.

Note that all chunks in an array have the same shape. If the length of any
array dimension is not exactly divisible by the length of the corresponding
chunk dimension then some chunks will overhang the edge of the array. The
contents of any chunk region falling outside the array are undefined.

.. _spec_v2_array_filters:

Filters
~~~~~~~

Optionally a sequence of one or more filters can be used to transform chunk
data prior to compression. When storing data, filters are applied in the order
specified in array metadata to encode data, then the encoded data are passed to
the primary compressor. When retrieving data, stored chunk data are
decompressed by the primary compressor then decoded using filters in the
reverse order.

.. _spec_v2_hierarchy:

Hierarchies
-----------

.. _spec_v2_hierarchy_paths:

Logical storage paths
~~~~~~~~~~~~~~~~~~~~~

Multiple arrays can be stored in the same array store by associating each array
with a different logical path. A logical path is simply an ASCII string. The
logical path is used to form a prefix for keys used by the array. For example,
if an array is stored at logical path "foo/bar" then the array metadata will be
stored under the key "foo/bar/.zarray", the user-defined attributes will be
stored under the key "foo/bar/.zattrs", and the chunks will be stored under
keys like "foo/bar/0.0", "foo/bar/0.1", etc.

To ensure consistent behaviour across different storage systems, logical paths
MUST be normalized as follows:

* Replace all backward slash characters ("\\\\") with forward slash characters
  ("/")
* Strip any leading "/" characters
* Strip any trailing "/" characters
* Collapse any sequence of more than one "/" character into a single "/"
  character

The key prefix is then obtained by appending a single "/" character to the
normalized logical path.

After normalization, if splitting a logical path by the "/" character results
in any path segment equal to the string "." or the string ".." then an error
MUST be raised.

N.B., how the underlying array store processes requests to store values under
keys containing the "/" character is entirely up to the store implementation
and is not constrained by this specification. E.g., an array store could simply
treat all keys as opaque ASCII strings; equally, an array store could map
logical paths onto some kind of hierarchical storage (e.g., directories on a
file system).

.. _spec_v2_hierarchy_groups:

Groups
~~~~~~

Arrays can be organized into groups which can also contain other groups. A
group is created by storing group metadata under the ".zgroup" key under some
logical path. E.g., a group exists at the root of an array store if the
".zgroup" key exists in the store, and a group exists at logical path "foo/bar"
if the "foo/bar/.zgroup" key exists in the store.

If the user requests a group to be created under some logical path, then groups
MUST also be created at all ancestor paths. E.g., if the user requests group
creation at path "foo/bar" then groups MUST be created at path "foo" and the
root of the store, if they don't already exist.

If the user requests an array to be created under some logical path, then
groups MUST also be created at all ancestor paths. E.g., if the user requests
array creation at path "foo/bar/baz" then groups must be created at path
"foo/bar", path "foo", and the root of the store, if they don't already exist.

The group metadata resource is a JSON object. The following keys MUST be present
within the object:

zarr_format
    An integer defining the version of the storage specification to which the
    array store adheres.

Other keys MUST NOT be present within the metadata object.

The members of a group are arrays and groups stored under logical paths that
are direct children of the parent group's logical path. E.g., if groups exist
under the logical paths "foo" and "foo/bar" and an array exists at logical path
"foo/baz" then the members of the group at path "foo" are the group at path
"foo/bar" and the array at path "foo/baz".

.. _spec_v2_attrs:

Attributes
----------

An array or group can be associated with custom attributes, which are arbitrary
key/value pairs with application-specific meaning. Custom attributes are encoded
as a JSON object and stored under the ".zattrs" key within an array store. The
".zattrs" key does not have to be present, and if it is absent the attributes
should be treated as empty.

For example, the JSON object below encodes three attributes named
"foo", "bar" and "baz"::

    {
        "foo": 42,
        "bar": "apples",
        "baz": [1, 2, 3, 4]
    }

.. _spec_v2_examples:

Examples
--------

Storing a single array
~~~~~~~~~~~~~~~~~~~~~~

Below is an example of storing a Zarr array, using a directory on the
local file system as storage.

Create an array::

    >>> import zarr
    >>> store = zarr.DirectoryStore('data/example.zarr')
    >>> a = zarr.create(shape=(20, 20), chunks=(10, 10), dtype='i4',
    ...                 fill_value=42, compressor=zarr.Zlib(level=1),
    ...                 store=store, overwrite=True)

No chunks are initialized yet, so only the ".zarray" and ".zattrs" keys
have been set in the store::

    >>> import os
    >>> sorted(os.listdir('data/example.zarr'))
    ['.zarray']

Inspect the array metadata::

    >>> print(open('data/example.zarr/.zarray').read())
    {
        "chunks": [
            10,
            10
        ],
        "compressor": {
            "id": "zlib",
            "level": 1
        },
        "dtype": "<i4",
        "fill_value": 42,
        "filters": null,
        "order": "C",
        "shape": [
            20,
            20
        ],
        "zarr_format": 2
    }

Chunks are initialized on demand. E.g., set some data::

    >>> a[0:10, 0:10] = 1
    >>> sorted(os.listdir('data/example.zarr'))
    ['.zarray', '0.0']

Set some more data::

    >>> a[0:10, 10:20] = 2
    >>> a[10:20, :] = 3
    >>> sorted(os.listdir('data/example.zarr'))
    ['.zarray', '0.0', '0.1', '1.0', '1.1']

Manually decompress a single chunk for illustration::

    >>> import zlib
    >>> buf = zlib.decompress(open('data/example.zarr/0.0', 'rb').read())
    >>> import numpy as np
    >>> chunk = np.frombuffer(buf, dtype='<i4')
    >>> chunk
    array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1], dtype=int32)

Modify the array attributes::

    >>> a.attrs['foo'] = 42
    >>> a.attrs['bar'] = 'apples'
    >>> a.attrs['baz'] = [1, 2, 3, 4]
    >>> sorted(os.listdir('data/example.zarr'))
    ['.zarray', '.zattrs', '0.0', '0.1', '1.0', '1.1']
    >>> print(open('data/example.zarr/.zattrs').read())
    {
        "bar": "apples",
        "baz": [
            1,
            2,
            3,
            4
        ],
        "foo": 42
    }

Storing multiple arrays in a hierarchy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Below is an example of storing multiple Zarr arrays organized into a group
hierarchy, using a directory on the local file system as storage. This storage
implementation maps logical paths onto directory paths on the file system,
however this is an implementation choice and is not required.

Setup the store::

    >>> import zarr
    >>> store = zarr.DirectoryStore('data/group.zarr')

Create the root group::

    >>> root_grp = zarr.group(store, overwrite=True)

The metadata resource for the root group has been created::

    >>> import os
    >>> sorted(os.listdir('data/group.zarr'))
    ['.zgroup']

Inspect the group metadata::

    >>> print(open('data/group.zarr/.zgroup').read())
    {
        "zarr_format": 2
    }

Create a sub-group::

    >>> sub_grp = root_grp.create_group('foo')

What has been stored::

    >>> sorted(os.listdir('data/group.zarr'))
    ['.zgroup', 'foo']
    >>> sorted(os.listdir('data/group.zarr/foo'))
    ['.zgroup']

Create an array within the sub-group::

    >>> a = sub_grp.create_dataset('bar', shape=(20, 20), chunks=(10, 10))
    >>> a[:] = 42

Set a custom attributes::

    >>> a.attrs['comment'] = 'answer to life, the universe and everything'

What has been stored::

    >>> sorted(os.listdir('data/group.zarr'))
    ['.zgroup', 'foo']
    >>> sorted(os.listdir('data/group.zarr/foo'))
    ['.zgroup', 'bar']
    >>> sorted(os.listdir('data/group.zarr/foo/bar'))
    ['.zarray', '.zattrs', '0.0', '0.1', '1.0', '1.1']

Here is the same example using a Zip file as storage::

    >>> store = zarr.ZipStore('data/group.zip', mode='w')
    >>> root_grp = zarr.group(store)
    >>> sub_grp = root_grp.create_group('foo')
    >>> a = sub_grp.create_dataset('bar', shape=(20, 20), chunks=(10, 10))
    >>> a[:] = 42
    >>> a.attrs['comment'] = 'answer to life, the universe and everything'
    >>> store.close()

What has been stored::

    >>> import zipfile
    >>> zf = zipfile.ZipFile('data/group.zip', mode='r')
    >>> for name in sorted(zf.namelist()):
    ...     print(name)
    .zgroup
    foo/.zgroup
    foo/bar/.zarray
    foo/bar/.zattrs
    foo/bar/0.0
    foo/bar/0.1
    foo/bar/1.0
    foo/bar/1.1

.. _spec_v2_changes:

Changes
-------

Version 2 clarifications
~~~~~~~~~~~~~~~~~~~~~~~~

The following changes have been made to the version 2 specification since it was
initially published to clarify ambiguities and add some missing information.

* The specification now describes how bytes fill values should be encoded and
  decoded for arrays with a fixed-length byte string data type (`#165
  <https://github.com/zarr-developers/zarr-python/issues/165/>`_, `#176
  <https://github.com/zarr-developers/zarr-python/issues/176/>`_).

* The specification now clarifies that units must be specified for datetime64 and
  timedelta64 data types (`#85
  <https://github.com/zarr-developers/zarr-python/issues/85/>`_, `#215
  <https://github.com/zarr-developers/zarr-python/issues/215/>`_).

* The specification now clarifies that the '.zattrs' key does not have to be present for
  either arrays or groups, and if absent then custom attributes should be treated as
  empty.

* The specification now describes how structured datatypes with
  subarray shapes and/or with nested structured data types are encoded
  in array metadata (`#111
  <https://github.com/zarr-developers/zarr-python/issues/111/>`_, `#296
  <https://github.com/zarr-developers/zarr-python/issues/296/>`_).

* Clarified the key/value pairs of custom attributes as "arbitrary" rather than
  "simple".

Changes from version 1 to version 2
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The following changes were made between version 1 and version 2 of this specification:

* Added support for storing multiple arrays in the same store and organising
  arrays into hierarchies using groups.
* Array metadata is now stored under the ".zarray" key instead of the "meta"
  key.
* Custom attributes are now stored under the ".zattrs" key instead of the
  "attrs" key.
* Added support for filters.
* Changed encoding of "fill_value" field within array metadata.
* Changed encoding of compressor information within array metadata to be
  consistent with representation of filter information.