Merge branch 'main' into PYL-W0404

Author: d-v-b

.github/workflows/hypothesis.yaml

@@ -87,7 +87,7 @@ jobs:
         && steps.status.outcome == 'failure'
         && github.event_name == 'schedule'
         && github.repository_owner == 'zarr-developers'
-      uses: xarray-contrib/issue-from-pytest-log@v1
+      uses: scientific-python/issue-from-pytest-log-action@v1
       with:
         log-path: output-${{ matrix.python-version }}-log.jsonl
         issue-title: "Nightly Hypothesis tests failed"

changes/2819.chore.rst

@@ -1,4 +1,18 @@
 Ensure that invocations of ``create_array`` use consistent keyword arguments, with consistent defaults.
-Specifically, ``zarr.api.synchronous.create_array`` now takes a ``write_data`` keyword argument; The
-``create_array`` method on ``zarr.Group`` takes ``data`` and ``write_data`` keyword arguments. The ``fill_value``
-keyword argument of the various invocations of ``create_array`` has been consistently set to ``None``, where previously it was either ``None`` or ``0``.
+
+- ``zarr.api.synchronous.create_array`` now takes a ``write_data`` keyword argument
+- The ``Group.create_array`` method takes ``data`` and ``write_data`` keyword arguments.
+- The functions ``api.asynchronous.create``, ``api.asynchronous.create_array``
+  and the methods ``Group.create_array``, ``Group.array``, had the default
+  ``fill_value`` changed from ``0`` to the ``DEFAULT_FILL_VALUE`` value, which instructs Zarr to
+  use the default scalar value associated with the array's data type as the fill value. These are
+  all functions or methods for array creation that mirror, wrap or are wrapped by, another function
+  that already has a default ``fill_value`` set to ``DEFAULT_FILL_VALUE``. This change is necessary
+  to make these functions consistent across the entire codebase, but as this changes default values,
+  new data might have a different fill value than expected after this change.
+
+  For data types where 0 is meaningful, like integers or floats, the default scalar is 0, so this
+  change should not be noticeable. For data types where 0 is ambiguous, like fixed-length unicode
+  strings, the default fill value might be different after this change. Users who were relying on how
+  Zarr interpreted ``0`` as a non-numeric scalar value should set their desired fill value explicitly
+  after this change.

changes/2874.feature.rst

@@ -1,9 +1,20 @@
-Adds zarr-specific data type classes. This replaces the internal use of numpy data types for zarr
-v2 and a fixed set of string enums for zarr v3. This change is largely internal, but it does
-change the type of the ``dtype`` and ``data_type`` fields on the ``ArrayV2Metadata`` and
-``ArrayV3Metadata`` classes. It also changes the JSON metadata representation of the
-variable-length string data type, but the old metadata representation can still be
-used when reading arrays. The logic for automatically choosing the chunk encoding for a given data
-type has also changed, and this necessitated changes to the ``config`` API.
+Adds zarr-specific data type classes.
+
+This change adds a ``ZDType`` base class for Zarr V2 and Zarr V3 data types. Child classes are
+defined for each NumPy data type. Each child class defines routines for ``JSON`` serialization.
+New data types can be created and registered dynamically.
+
+Prior to this change, Zarr Python had two streams for handling data types. For Zarr V2 arrays,
+we used NumPy data type identifiers. For Zarr V3 arrays, we used a fixed set of string enums. Both
+of these systems proved hard to extend.
+
+This change is largely internal, but it does change the type of the ``dtype`` and ``data_type``
+fields on the ``ArrayV2Metadata`` and ``ArrayV3Metadata`` classes. Previously, ``ArrayV2Metadata.dtype``
+was a NumPy ``dtype`` object, and ``ArrayV3Metadata.data_type`` was an internally-defined ``enum``.
+After this change, both ``ArrayV2Metadata.dtype`` and ``ArrayV3Metadata.data_type`` are instances of
+``ZDType``. A NumPy data type can be generated from a ``ZDType`` via the ``ZDType.to_native_dtype()``
+method. The internally-defined Zarr V3 ``enum`` class is gone entirely, but the ``ZDType.to_json(zarr_format=3)``
+method can be used to generate either a string, or dictionary that has a string ``name`` field, that
+represents the string value previously associated with that ``enum``.
 
 For more on this new feature, see the `documentation </user-guide/data_types.html>`_

changes/3233.feature.rst

@@ -0,0 +1 @@
+Add an alternate `from_array_metadata_and_store` constructor to `CodecPipeline`.

src/zarr/abc/codec.py

@@ -12,11 +12,12 @@
     from collections.abc import Awaitable, Callable, Iterable
     from typing import Self
 
-    from zarr.abc.store import ByteGetter, ByteSetter
+    from zarr.abc.store import ByteGetter, ByteSetter, Store
     from zarr.core.array_spec import ArraySpec
     from zarr.core.chunk_grids import ChunkGrid
     from zarr.core.dtype.wrapper import TBaseDType, TBaseScalar, ZDType
     from zarr.core.indexing import SelectorTuple
+    from zarr.core.metadata import ArrayMetadata
 
 __all__ = [
     "ArrayArrayCodec",
@@ -281,6 +282,25 @@ def from_codecs(cls, codecs: Iterable[Codec]) -> Self:
         """
         ...
 
+    @classmethod
+    def from_array_metadata_and_store(cls, array_metadata: ArrayMetadata, store: Store) -> Self:
+        """Creates a codec pipeline from array metadata and a store path.
+
+        Raises NotImplementedError by default, indicating the CodecPipeline must be created with from_codecs instead.
+
+        Parameters
+        ----------
+        array_metadata : ArrayMetadata
+        store : Store
+
+        Returns
+        -------
+        Self
+        """
+        raise NotImplementedError(
+            f"'{type(cls).__name__}' does not implement CodecPipeline.from_array_metadata_and_store."
+        )
+
     @property
     @abstractmethod
     def supports_partial_decode(self) -> bool: ...

src/zarr/api/synchronous.py

@@ -6,7 +6,6 @@
 
 import zarr.api.asynchronous as async_api
 import zarr.core.array
-from zarr._compat import _deprecate_positional_args
 from zarr.core.array import DEFAULT_FILL_VALUE, Array, AsyncArray, CompressorLike
 from zarr.core.group import Group
 from zarr.core.sync import sync
@@ -160,7 +159,6 @@ def load(
     )
 
 
-@_deprecate_positional_args
 def open(
     store: StoreLike | None = None,
     *,
@@ -255,7 +253,6 @@ def save(
     )
 
 
-@_deprecate_positional_args
 def save_array(
     store: StoreLike,
     arr: NDArrayLike,
@@ -387,7 +384,6 @@ def array(data: npt.ArrayLike | Array, **kwargs: Any) -> Array:
     return Array(sync(async_api.array(data=data, **kwargs)))
 
 
-@_deprecate_positional_args
 def group(
     store: StoreLike | None = None,
     *,
@@ -455,7 +451,6 @@ def group(
     )
 
 
-@_deprecate_positional_args
 def open_group(
     store: StoreLike | None = None,
     *,

src/zarr/core/array.py

@@ -25,7 +25,6 @@
 from typing_extensions import deprecated
 
 import zarr
-from zarr._compat import _deprecate_positional_args
 from zarr.abc.codec import ArrayArrayCodec, ArrayBytesCodec, BytesBytesCodec, Codec
 from zarr.abc.store import Store, set_or_delete
 from zarr.codecs._v2 import V2Codec
@@ -192,7 +191,15 @@ def parse_array_metadata(data: Any) -> ArrayMetadata:
     raise TypeError  # pragma: no cover
 
 
-def create_codec_pipeline(metadata: ArrayMetadata) -> CodecPipeline:
+def create_codec_pipeline(metadata: ArrayMetadata, *, store: Store | None = None) -> CodecPipeline:
+    if store is not None:
+        try:
+            return get_pipeline_class().from_array_metadata_and_store(
+                array_metadata=metadata, store=store
+            )
+        except NotImplementedError:
+            pass
+
     if isinstance(metadata, ArrayV3Metadata):
         return get_pipeline_class().from_codecs(metadata.codecs)
     elif isinstance(metadata, ArrayV2Metadata):
@@ -311,7 +318,11 @@ def __init__(
         object.__setattr__(self, "metadata", metadata_parsed)
         object.__setattr__(self, "store_path", store_path)
         object.__setattr__(self, "_config", config_parsed)
-        object.__setattr__(self, "codec_pipeline", create_codec_pipeline(metadata=metadata_parsed))
+        object.__setattr__(
+            self,
+            "codec_pipeline",
+            create_codec_pipeline(metadata=metadata_parsed, store=store_path.store),
+        )
 
     # this overload defines the function signature when zarr_format is 2
     @overload
@@ -430,7 +441,6 @@ async def create(
 
     @classmethod
     @deprecated("Use zarr.api.asynchronous.create_array instead.")
-    @_deprecate_positional_args
     async def create(
         cls,
         store: StoreLike,
@@ -1782,7 +1792,6 @@ class Array:
 
     @classmethod
     @deprecated("Use zarr.create_array instead.")
-    @_deprecate_positional_args
     def create(
         cls,
         store: StoreLike,
@@ -2595,7 +2604,6 @@ def __setitem__(self, selection: Selection, value: npt.ArrayLike) -> None:
         else:
             self.set_basic_selection(cast("BasicSelection", pure_selection), value, fields=fields)
 
-    @_deprecate_positional_args
     def get_basic_selection(
         self,
         selection: BasicSelection = Ellipsis,
@@ -2719,7 +2727,6 @@ def get_basic_selection(
             )
         )
 
-    @_deprecate_positional_args
     def set_basic_selection(
         self,
         selection: BasicSelection,
@@ -2815,7 +2822,6 @@ def set_basic_selection(
         indexer = BasicIndexer(selection, self.shape, self.metadata.chunk_grid)
         sync(self._async_array._set_selection(indexer, value, fields=fields, prototype=prototype))
 
-    @_deprecate_positional_args
     def get_orthogonal_selection(
         self,
         selection: OrthogonalSelection,
@@ -2940,7 +2946,6 @@ def get_orthogonal_selection(
             )
         )
 
-    @_deprecate_positional_args
     def set_orthogonal_selection(
         self,
         selection: OrthogonalSelection,
@@ -3051,7 +3056,6 @@ def set_orthogonal_selection(
             self._async_array._set_selection(indexer, value, fields=fields, prototype=prototype)
         )
 
-    @_deprecate_positional_args
     def get_mask_selection(
         self,
         mask: MaskSelection,
@@ -3134,7 +3138,6 @@ def get_mask_selection(
             )
         )
 
-    @_deprecate_positional_args
     def set_mask_selection(
         self,
         mask: MaskSelection,
@@ -3213,7 +3216,6 @@ def set_mask_selection(
         indexer = MaskIndexer(mask, self.shape, self.metadata.chunk_grid)
         sync(self._async_array._set_selection(indexer, value, fields=fields, prototype=prototype))
 
-    @_deprecate_positional_args
     def get_coordinate_selection(
         self,
         selection: CoordinateSelection,
@@ -3303,7 +3305,6 @@ def get_coordinate_selection(
             out_array = np.array(out_array).reshape(indexer.sel_shape)
         return out_array
 
-    @_deprecate_positional_args
     def set_coordinate_selection(
         self,
         selection: CoordinateSelection,
@@ -3401,7 +3402,6 @@ def set_coordinate_selection(
 
         sync(self._async_array._set_selection(indexer, value, fields=fields, prototype=prototype))
 
-    @_deprecate_positional_args
     def get_block_selection(
         self,
         selection: BasicSelection,
@@ -3500,7 +3500,6 @@ def get_block_selection(
             )
         )
 
-    @_deprecate_positional_args
     def set_block_selection(
         self,
         selection: BasicSelection,

src/zarr/core/group.py

@@ -15,7 +15,6 @@
 from typing_extensions import deprecated
 
 import zarr.api.asynchronous as async_api
-from zarr._compat import _deprecate_positional_args
 from zarr.abc.metadata import Metadata
 from zarr.abc.store import Store, set_or_delete
 from zarr.core._info import GroupInfo
@@ -2417,7 +2416,6 @@ def create(self, *args: Any, **kwargs: Any) -> Array:
         # Backwards compatibility for 2.x
         return self.create_array(*args, **kwargs)
 
-    @_deprecate_positional_args
     def create_array(
         self,
         name: str,
@@ -2635,7 +2633,6 @@ def require_array(self, name: str, *, shape: ShapeLike, **kwargs: Any) -> Array:
         """
         return Array(self._sync(self._async_group.require_array(name, shape=shape, **kwargs)))
 
-    @_deprecate_positional_args
     def empty(self, *, name: str, shape: ChunkCoords, **kwargs: Any) -> Array:
         """Create an empty array with the specified shape in this Group. The contents will be filled with
         the array's fill value or zeros if no fill value is provided.
@@ -2657,7 +2654,6 @@ def empty(self, *, name: str, shape: ChunkCoords, **kwargs: Any) -> Array:
         """
         return Array(self._sync(self._async_group.empty(name=name, shape=shape, **kwargs)))
 
-    @_deprecate_positional_args
     def zeros(self, *, name: str, shape: ChunkCoords, **kwargs: Any) -> Array:
         """Create an array, with zero being used as the default value for uninitialized portions of the array.
 
@@ -2677,7 +2673,6 @@ def zeros(self, *, name: str, shape: ChunkCoords, **kwargs: Any) -> Array:
         """
         return Array(self._sync(self._async_group.zeros(name=name, shape=shape, **kwargs)))
 
-    @_deprecate_positional_args
     def ones(self, *, name: str, shape: ChunkCoords, **kwargs: Any) -> Array:
         """Create an array, with one being used as the default value for uninitialized portions of the array.
 
@@ -2697,7 +2692,6 @@ def ones(self, *, name: str, shape: ChunkCoords, **kwargs: Any) -> Array:
         """
         return Array(self._sync(self._async_group.ones(name=name, shape=shape, **kwargs)))
 
-    @_deprecate_positional_args
     def full(
         self, *, name: str, shape: ChunkCoords, fill_value: Any | None, **kwargs: Any
     ) -> Array:
@@ -2725,7 +2719,6 @@ def full(
             )
         )
 
-    @_deprecate_positional_args
     def empty_like(self, *, name: str, data: async_api.ArrayLike, **kwargs: Any) -> Array:
         """Create an empty sub-array like `data`. The contents will be filled
         with the array's fill value or zeros if no fill value is provided.
@@ -2752,7 +2745,6 @@ def empty_like(self, *, name: str, data: async_api.ArrayLike, **kwargs: Any) ->
         """
         return Array(self._sync(self._async_group.empty_like(name=name, data=data, **kwargs)))
 
-    @_deprecate_positional_args
     def zeros_like(self, *, name: str, data: async_api.ArrayLike, **kwargs: Any) -> Array:
         """Create a sub-array of zeros like `data`.
 
@@ -2773,7 +2765,6 @@ def zeros_like(self, *, name: str, data: async_api.ArrayLike, **kwargs: Any) ->
 
         return Array(self._sync(self._async_group.zeros_like(name=name, data=data, **kwargs)))
 
-    @_deprecate_positional_args
     def ones_like(self, *, name: str, data: async_api.ArrayLike, **kwargs: Any) -> Array:
         """Create a sub-array of ones like `data`.
 
@@ -2793,7 +2784,6 @@ def ones_like(self, *, name: str, data: async_api.ArrayLike, **kwargs: Any) -> A
         """
         return Array(self._sync(self._async_group.ones_like(name=name, data=data, **kwargs)))
 
-    @_deprecate_positional_args
     def full_like(self, *, name: str, data: async_api.ArrayLike, **kwargs: Any) -> Array:
         """Create a sub-array like `data` filled with the `fill_value` of `data` .
 
@@ -2823,7 +2813,6 @@ def move(self, source: str, dest: str) -> None:
         return self._sync(self._async_group.move(source, dest))
 
     @deprecated("Use Group.create_array instead.")
-    @_deprecate_positional_args
     def array(
         self,
         name: str,