Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accept dictionaries for store argument #2164

Merged
merged 3 commits into from
Sep 12, 2024

Conversation

TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Sep 9, 2024

This updates how we handle a plain dictionary provided to open_group and friends so that the following works:

group = await zarr.api.asynchronous.open_group(store={}, mode="w")

The dictionary is interpreted as the store_dict argument of zarr.store.MemoryStore. This effectively mirrors the behavior of zarr-python v2, and I think doesn't introduce an ambiguity. Previously a TypeError was raised.

This change will make it slightly easier to get xarray's test suite running with zarr v3.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
Copy link
Member

jhamman commented Sep 11, 2024

After thinking a bit more about this, I'm a bit concerned that we may not be ready for a generic MutableMapping in the MemoryStore. What would happen if we tossed a zict.LMDB into the constructor? I suspect things would mostly work but maybe not?

@TomAugspurger
Copy link
Contributor Author

I'm not sure how much faith to put in it, but the fact that mypy allows it gives me some comfort.

@d-v-b
Copy link
Contributor

d-v-b commented Sep 11, 2024

If we accept anything that implements MutableMapping here, then there's no guarantee that the store is actually in memory, since the __getitem__ and __setitem__ operations could be running arbitrary IO-ful code. This seems fine with me, as long as users know what they are getting into, and maybe a useful bridge for users who want some v2-style store behavior (but maybe a better name for the store would be MutableMappingStore?)

@TomAugspurger
Copy link
Contributor Author

Ah, I see, the concern is specifically around the isinstance at https://github.com/zarr-developers/zarr-python/pull/2164/files#diff-9a2583a58394683284176a2d2db4df1e8ffafce4f34edc7a176c625863815518R99.

In that case, yes, I think we should narrow it to just accept dictionaries, where there's no ambiguity about whether a MemoryStore is appropriate.

@jhamman jhamman merged commit c62294e into zarr-developers:v3 Sep 12, 2024
26 checks passed
dcherian added a commit to dcherian/zarr-python that referenced this pull request Sep 16, 2024
* v3:
  fix: opening a group with unspecified format finds either v2 or v3 (zarr-developers#2183)
  test: check that store, array, and group classes are serializable  (zarr-developers#2006)
  feature(store): V3 ZipStore (zarr-developers#2078)
  More typing fixes for tests (zarr-developers#2173)
  refactor: split metadata into v2 and v3 modules (zarr-developers#2163)
  Accept dictionaries for `store` argument (zarr-developers#2164)
  Simplify mypy config for tests (zarr-developers#2156)
  Fixed path segment duplication in open_array (zarr-developers#2167)
  Fixed test warnings (zarr-developers#2168)
  chore: update pre-commit hooks (zarr-developers#2165)
  Ensure that store_dict used for empty dicts (zarr-developers#2162)
  Bump pypa/gh-action-pypi-publish from 1.10.0 to 1.10.1 in the actions group (zarr-developers#2160)
dstansby pushed a commit to dstansby/zarr-python that referenced this pull request Sep 17, 2024
* Accept dictionaries for `store`

* fixup doc build warnings

* Only support dictionaries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants