Merge branch 'main' into datatree_alignment_docs

.github/FUNDING.yml

@@ -1,2 +1,2 @@
 github: numfocus
-custom: http://numfocus.org/donate-to-xarray
+custom: https://numfocus.org/donate-to-xarray

.github/workflows/ci.yaml

@@ -58,6 +58,9 @@ jobs:
             python-version: "3.10"
             os: ubuntu-latest
           # Latest python version:
+          - env: "all-but-numba"
+            python-version: "3.12"
+            os: ubuntu-latest
           - env: "all-but-dask"
             # Not 3.12 because of pint
             python-version: "3.11"

.github/workflows/nightly-wheels.yml

@@ -13,7 +13,7 @@ jobs:
           fetch-depth: 0
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.11"
+          python-version: "3.12"
 
       - name: Install dependencies
         run: |
@@ -38,7 +38,7 @@ jobs:
           fi
 
       - name: Upload wheel
-        uses: scientific-python/upload-nightly-action@b67d7fcc0396e1128a474d1ab2b48aa94680f9fc # 0.5.0
+        uses: scientific-python/upload-nightly-action@82396a2ed4269ba06c6b2988bb4fd568ef3c3d6b # 0.6.1
         with:
           anaconda_nightly_upload_token: ${{ secrets.ANACONDA_NIGHTLY }}
           artifacts_path: dist

README.md

@@ -88,7 +88,7 @@ Xarray is a fiscally sponsored project of
 [NumFOCUS](https://numfocus.org), a nonprofit dedicated to supporting
 the open source scientific computing community. If you like Xarray and
 want to support our mission, please consider making a
-[donation](https://numfocus.salsalabs.org/donate-to-xarray/) to support
+[donation](https://numfocus.org/donate-to-xarray) to support
 our efforts.
 
 ## History

ci/requirements/all-but-numba.yml

@@ -0,0 +1,54 @@
+name: xarray-tests
+channels:
+  - conda-forge
+  - nodefaults
+dependencies:
+  # Pin a "very new numpy" (updated Sept 24, 2024)
+  - numpy>=2.1.1
+  - aiobotocore
+  - array-api-strict
+  - boto3
+  - bottleneck
+  - cartopy
+  - cftime
+  - dask-core
+  - dask-expr # dask raises a deprecation warning without this, breaking doctests
+  - distributed
+  - flox
+  - fsspec
+  - h5netcdf
+  - h5py
+  - hdf5
+  - hypothesis
+  - iris
+  - lxml # Optional dep of pydap
+  - matplotlib-base
+  - nc-time-axis
+  - netcdf4
+  # numba, sparse, numbagg, numexpr often conflicts with newer versions of numpy.
+  # This environment helps us test xarray with the latest versions
+  # of numpy
+  # - numba
+  # - numbagg
+  # - numexpr
+  # - sparse
+  - opt_einsum
+  - packaging
+  - pandas
+  # - pint>=0.22
+  - pip
+  - pooch
+  - pre-commit
+  - pyarrow # pandas raises a deprecation warning without this, breaking doctests
+  - pydap
+  - pytest
+  - pytest-cov
+  - pytest-env
+  - pytest-xdist
+  - pytest-timeout
+  - rasterio
+  - scipy
+  - seaborn
+  - toolz
+  - typing_extensions
+  - zarr

doc/ecosystem.rst

@@ -26,7 +26,7 @@ Geosciences
 - `OGGM <https://oggm.org/>`_: Open Global Glacier Model
 - `Oocgcm <https://oocgcm.readthedocs.io/>`_: Analysis of large gridded geophysical datasets
 - `Open Data Cube <https://www.opendatacube.org/>`_: Analysis toolkit of continental scale Earth Observation data from satellites.
-- `Pangaea: <https://pangaea.readthedocs.io/en/latest/>`_: xarray extension for gridded land surface & weather model output).
+- `Pangaea <https://pangaea.readthedocs.io/en/latest/>`_: xarray extension for gridded land surface & weather model output).
 - `Pangeo <https://pangeo.io>`_: A community effort for big data geoscience in the cloud.
 - `PyGDX <https://pygdx.readthedocs.io/en/latest/>`_: Python 3 package for
   accessing data stored in GAMS Data eXchange (GDX) files. Also uses a custom

doc/howdoi.rst

@@ -58,7 +58,7 @@ How do I ...
    * - apply a function on all data variables in a Dataset
      - :py:meth:`Dataset.map`
    * - write xarray objects with complex values to a netCDF file
-     - :py:func:`Dataset.to_netcdf`, :py:func:`DataArray.to_netcdf` specifying ``engine="h5netcdf", invalid_netcdf=True``
+     - :py:func:`Dataset.to_netcdf`, :py:func:`DataArray.to_netcdf` specifying ``engine="h5netcdf"`` or :py:func:`Dataset.to_netcdf`, :py:func:`DataArray.to_netcdf` specifying ``engine="netCDF4", auto_complex=True``
    * - make xarray objects look like other xarray objects
      - :py:func:`~xarray.ones_like`, :py:func:`~xarray.zeros_like`, :py:func:`~xarray.full_like`, :py:meth:`Dataset.reindex_like`, :py:meth:`Dataset.interp_like`, :py:meth:`Dataset.broadcast_like`, :py:meth:`DataArray.reindex_like`, :py:meth:`DataArray.interp_like`, :py:meth:`DataArray.broadcast_like`
    * - Make sure my datasets have values at the same coordinate locations

doc/user-guide/io.rst

@@ -566,29 +566,12 @@ This is not CF-compliant but again facilitates roundtripping of xarray datasets.
 Invalid netCDF files
 ~~~~~~~~~~~~~~~~~~~~
 
-The library ``h5netcdf`` allows writing some dtypes (booleans, complex, ...) that aren't
+The library ``h5netcdf`` allows writing some dtypes that aren't
 allowed in netCDF4 (see
-`h5netcdf documentation <https://github.com/shoyer/h5netcdf#invalid-netcdf-files>`_).
+`h5netcdf documentation <https://github.com/h5netcdf/h5netcdf#invalid-netcdf-files>`_).
 This feature is available through :py:meth:`DataArray.to_netcdf` and
 :py:meth:`Dataset.to_netcdf` when used with ``engine="h5netcdf"``
-and currently raises a warning unless ``invalid_netcdf=True`` is set:
-
-.. ipython:: python
-    :okwarning:
-
-    # Writing complex valued data
-    da = xr.DataArray([1.0 + 1.0j, 2.0 + 2.0j, 3.0 + 3.0j])
-    da.to_netcdf("complex.nc", engine="h5netcdf", invalid_netcdf=True)
-
-    # Reading it back
-    reopened = xr.open_dataarray("complex.nc", engine="h5netcdf")
-    reopened
-
-.. ipython:: python
-    :suppress:
-
-    reopened.close()
-    os.remove("complex.nc")
+and currently raises a warning unless ``invalid_netcdf=True`` is set.
 
 .. warning::
 

doc/user-guide/pandas.rst

@@ -103,7 +103,7 @@ DataFrames:
     xr.DataArray.from_series(s)
 
 Both the ``from_series`` and ``from_dataframe`` methods use reindexing, so they
-work even if not the hierarchical index is not a full tensor product:
+work even if the hierarchical index is not a full tensor product:
 
 .. ipython:: python
 
@@ -126,7 +126,7 @@ Particularly after a roundtrip, the following deviations are noted:
 To avoid these problems, the third-party `ntv-pandas <https://github.com/loco-philippe/ntv-pandas>`__ library offers lossless and reversible conversions between
 ``Dataset``/ ``DataArray`` and pandas ``DataFrame`` objects.
 
-This solution is particularly interesting for converting any ``DataFrame`` into a ``Dataset`` (the converter find the multidimensional structure hidden by the tabular structure).
+This solution is particularly interesting for converting any ``DataFrame`` into a ``Dataset`` (the converter finds the multidimensional structure hidden by the tabular structure).
 
 The `ntv-pandas examples <https://github.com/loco-philippe/ntv-pandas/tree/main/example>`__ show how to improve the conversion for the previous ``Dataset`` example and for more complex examples.
 

doc/whats-new.rst

@@ -32,6 +32,11 @@ New Features
   `Tom Nicholas <https://github.com/TomNicholas>`_.
 - Added zarr backends for :py:func:`open_groups` (:issue:`9430`, :pull:`9469`).
   By `Eni Awowale <https://github.com/eni-awowale>`_.
+- Added support for vectorized interpolation using additional interpolators
+  from the ``scipy.interpolate`` module (:issue:`9049`, :pull:`9526`).
+  By `Holly Mandel <https://github.com/hollymandel>`_.
+- Implement handling of complex numbers (netcdf4/h5netcdf) and enums (h5netcdf) (:issue:`9246`, :issue:`3297`, :pull:`9509`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
 
 Breaking changes
 ~~~~~~~~~~~~~~~~
@@ -59,7 +64,7 @@ Bug fixes
 - Fix a few bugs affecting groupby reductions with `flox`. (:issue:`8090`, :issue:`9398`).
   By `Deepak Cherian <https://github.com/dcherian>`_.
 - Fix the safe_chunks validation option on the to_zarr method
-  (:issue:`5511`, :pull:`9513`). By `Joseph Nowak
+  (:issue:`5511`, :pull:`9559`). By `Joseph Nowak
   <https://github.com/josephnowak>`_.
 
 Documentation
@@ -150,6 +155,9 @@ Bug fixes
 - Fix deprecation warning that was raised when calling ``np.array`` on an ``xr.DataArray``
   in NumPy 2.0 (:issue:`9312`, :pull:`9393`)
   By `Andrew Scherer <https://github.com/andrew-s28>`_.
+- Fix passing missing arguments to when opening hdf5 and netCDF4 datatrees
+  (:issue:`9427`, :pull: `9428`).
+  By `Alfonso Ladino <https://github.com/aladinor>`_.
 - Fix support for using ``pandas.DateOffset``, ``pandas.Timedelta``, and
   ``datetime.timedelta`` objects as ``resample`` frequencies
   (:issue:`9408`, :pull:`9413`).

pyproject.toml

@@ -27,9 +27,14 @@ dependencies = [
   "pandas>=2.1",
 ]
 
+# We don't encode minimum requirements here (though if we can write a script to
+# generate the text from `min_deps_check.py`, that's welcome...). We do add
+# `numba>=0.54` here because of https://github.com/astral-sh/uv/issues/7881;
+# note that it's not a direct dependency of xarray.
+
 [project.optional-dependencies]
-accel = ["scipy", "bottleneck", "numbagg", "flox", "opt_einsum"]
-complete = ["xarray[accel,io,parallel,viz,dev]"]
+accel = ["scipy", "bottleneck", "numbagg", "numba>=0.54", "flox", "opt_einsum"]
+complete = ["xarray[accel,etc,io,parallel,viz]"]
 dev = [
   "hypothesis",
   "mypy",
@@ -40,11 +45,14 @@ dev = [
   "pytest-xdist",
   "pytest-timeout",
   "ruff",
+  "sphinx",
+  "sphinx_autosummary_accessors",
   "xarray[complete]",
 ]
 io = ["netCDF4", "h5netcdf", "scipy", 'pydap; python_version<"3.10"', "zarr<3", "fsspec", "cftime", "pooch"]
+etc = ["sparse"]
 parallel = ["dask[complete]"]
-viz = ["matplotlib", "seaborn", "nc-time-axis"]
+viz = ["cartopy", "matplotlib", "nc-time-axis", "seaborn"]
 
 [project.urls]
 Documentation = "https://docs.xarray.dev"
@@ -206,32 +214,16 @@ warn_return_any = true
 
 module = ["xarray.namedarray.*", "xarray.tests.test_namedarray"]
 
-[tool.pyright]
-# include = ["src"]
-# exclude = ["**/node_modules",
-# "**/__pycache__",
-# "src/experimental",
-# "src/typestubs"
-# ]
-# ignore = ["src/oldstuff"]
-defineConstant = {DEBUG = true}
-# stubPath = "src/stubs"
-# venv = "env367"
-
-# Enabling this means that developers who have disabled the warning locally —
-# because not all dependencies are installable — are overridden
-# reportMissingImports = true
-reportMissingTypeStubs = false
-
-# pythonVersion = "3.6"
-# pythonPlatform = "Linux"
-
-# executionEnvironments = [
-# { root = "src/web", pythonVersion = "3.5", pythonPlatform = "Windows", extraPaths = [ "src/service_libs" ] },
-# { root = "src/sdk", pythonVersion = "3.0", extraPaths = [ "src/backend" ] },
-# { root = "src/tests", extraPaths = ["src/tests/e2e", "src/sdk" ]},
-# { root = "src" }
-# ]
+# We disable pyright here for now, since including it means that all errors show
+# up in devs' VS Code, which then makes it more difficult to work with actual
+# errors. It overrides local VS Code settings so isn't escapable.
+
+# [tool.pyright]
+# defineConstant = {DEBUG = true}
+# # Enabling this means that developers who have disabled the warning locally —
+# # because not all dependencies are installable — are overridden
+# # reportMissingImports = true
+# reportMissingTypeStubs = false
 
 [tool.ruff]
 extend-exclude = [
@@ -323,7 +315,6 @@ filterwarnings = [
   "default:Using a non-tuple sequence for multidimensional indexing is deprecated:FutureWarning",
   "default:Duplicate dimension names present:UserWarning:xarray.namedarray.core",
   "default:::xarray.tests.test_strategies", # TODO: remove once we know how to deal with a changed signature in protocols
-  "ignore:__array__ implementation doesn't accept a copy keyword, so passing copy=False failed.",
 ]
 
 log_cli_level = "INFO"

xarray/backends/api.py

@@ -1213,6 +1213,7 @@ def to_netcdf(
     *,
     multifile: Literal[True],
     invalid_netcdf: bool = False,
+    auto_complex: bool | None = None,
 ) -> tuple[ArrayWriter, AbstractDataStore]: ...
 
 
@@ -1230,6 +1231,7 @@ def to_netcdf(
     compute: bool = True,
     multifile: Literal[False] = False,
     invalid_netcdf: bool = False,
+    auto_complex: bool | None = None,
 ) -> bytes: ...
 
 
@@ -1248,6 +1250,7 @@ def to_netcdf(
     compute: Literal[False],
     multifile: Literal[False] = False,
     invalid_netcdf: bool = False,
+    auto_complex: bool | None = None,
 ) -> Delayed: ...
 
 
@@ -1265,6 +1268,7 @@ def to_netcdf(
     compute: Literal[True] = True,
     multifile: Literal[False] = False,
     invalid_netcdf: bool = False,
+    auto_complex: bool | None = None,
 ) -> None: ...
 
 
@@ -1283,6 +1287,7 @@ def to_netcdf(
     compute: bool = False,
     multifile: Literal[False] = False,
     invalid_netcdf: bool = False,
+    auto_complex: bool | None = None,
 ) -> Delayed | None: ...
 
 
@@ -1301,6 +1306,7 @@ def to_netcdf(
     compute: bool = False,
     multifile: bool = False,
     invalid_netcdf: bool = False,
+    auto_complex: bool | None = None,
 ) -> tuple[ArrayWriter, AbstractDataStore] | Delayed | None: ...
 
 
@@ -1318,6 +1324,7 @@ def to_netcdf(
     compute: bool = False,
     multifile: bool = False,
     invalid_netcdf: bool = False,
+    auto_complex: bool | None = None,
 ) -> tuple[ArrayWriter, AbstractDataStore] | bytes | Delayed | None: ...
 
 
@@ -1333,6 +1340,7 @@ def to_netcdf(
     compute: bool = True,
     multifile: bool = False,
     invalid_netcdf: bool = False,
+    auto_complex: bool | None = None,
 ) -> tuple[ArrayWriter, AbstractDataStore] | bytes | Delayed | None:
     """This function creates an appropriate datastore for writing a dataset to
     disk as a netCDF file
@@ -1400,6 +1408,9 @@ def to_netcdf(
             raise ValueError(
                 f"unrecognized option 'invalid_netcdf' for engine {engine}"
             )
+    if auto_complex is not None:
+        kwargs["auto_complex"] = auto_complex
+
     store = store_open(target, mode, format, group, **kwargs)
 
     if unlimited_dims is None: