[do not merge] zarrs bindings

.github/workflows/engine.yml

@@ -0,0 +1,49 @@
+name: Array engine
+
+on:
+  push:
+    branches: [ main ]
+    paths:
+      - 'src/zarr/**'
+      - 'tests/engine/**'
+      - 'tests/zarrista/**'
+      - 'pyproject.toml'
+      - '.github/workflows/engine.yml'
+  pull_request:
+    branches: [ main ]
+    paths:
+      - 'src/zarr/**'
+      - 'tests/engine/**'
+      - 'tests/zarrista/**'
+      - 'pyproject.toml'
+      - '.github/workflows/engine.yml'
+  workflow_dispatch:
+
+permissions: {}
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          fetch-depth: 0  # hatch-vcs needs tags to compute zarr's version
+          persist-credentials: false
+      - name: Install Rust
+        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
+      - name: Install uv
+        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+        with:
+          python-version: '3.12'
+      - name: Sync zarrista group
+        run: uv sync --group zarrista
+      - name: Run engine tests
+        # the ubuntu runner image ships a Rust toolchain; the maturin build
+        # backend for zarrista is fetched by uv on demand
+        run: uv run --group zarrista pytest tests/engine tests/zarrista -v

changes/4181.feature.md

@@ -0,0 +1,21 @@
+`Array` and `AsyncArray` now route data I/O through pluggable *array engines*
+(`zarr.abc.engine.ArrayEngine` / `AsyncArrayEngine`). An engine is selected
+with `engine=` on `zarr.create_array`, `zarr.open_array`, `Group.create_array`
+and the other array entry points; `zarr.list_engines()` lists the built-in
+names.
+
+The default engine is the existing codec-pipeline path, so behavior and
+performance are unchanged when `engine=` is omitted.
+
+Pass `engine="zarrista"` (requires the `zarrista` package) to serve Zarr v3
+arrays through the Rust `zarrs` implementation, on local-filesystem, obstore,
+or icechunk storage. It supports basic, orthogonal, coordinate, mask and block
+selections, for both reads and writes.
+
+An engine receives the user's selection as a `SelectionRequest`, which carries
+the selection unresolved along with the dialect it is written in. A backend
+that reads chunks takes the `Indexer` the request builds on demand; a backend
+that reads boxes — an FFI binding, an HTTP range endpoint — resolves the
+selection through [zarr-indexing](https://zarr-indexing.readthedocs.io/), which
+grafts the full NumPy indexing dialect onto a source offering only step-1
+slices. The zarrista engine takes the latter route.

docs/api/zarr/abc/engine.md

@@ -0,0 +1,5 @@
+---
+title: engine
+---
+
+::: zarr.abc.engine

docs/api/zarr/zarrista.md

@@ -0,0 +1,5 @@
+---
+title: zarrista
+---
+
+::: zarr.zarrista

docs/user-guide/examples/open_with_engine.md

@@ -0,0 +1,7 @@
+--8<-- "examples/open_with_engine/README.md"
+
+## Source Code
+
+```python exec="false" reason="pymdownx snippet include directive, not python source"
+--8<-- "examples/open_with_engine/open_with_engine.py"
+```

examples/open_with_engine/README.md

@@ -0,0 +1,51 @@
+# Open With a Different Engine Example
+
+This example demonstrates how to open the **same array data with a different
+backend** -- what zarr-python calls an *engine*.
+
+An engine is purely an *execution* setting: it selects which compute backend
+reads and writes an array's chunks. The bytes on disk are identical regardless
+of the engine, so the same Zarr array can be driven by a different backend
+without rewriting any data.
+
+The example shows how to:
+
+- Discover the available engines with `zarr.list_engines()`.
+- Inspect the engine an array is using via its public `array.engine` property.
+- Write one array once and read it back through several engines, asserting the
+  results are byte-for-byte identical (`numpy.testing.assert_array_equal`).
+- Select an engine **per call** via the `engine=` kwarg on `zarr.open_array`
+  and `zarr.create_array`.
+- Use the `"default"` engine, which works on any store and format.
+- Use the `"zarrista"` (Rust-backed) engine, which serves Zarr v3 arrays on a
+  `LocalStore` or an obstore-backed `ObjectStore` (the example guards this so it
+  still runs if the package is absent).
+
+## Available engines
+
+| Engine       | Backend           | Stores                              | Notes                           |
+| ------------ | ----------------- | ----------------------------------- | ------------------------------- |
+| `"default"`  | built-in Python   | any                                 | used when `engine=` is omitted  |
+| `"zarrista"` | Rust (`zarrista`) | `LocalStore`, obstore `ObjectStore` | requires the `zarrista` package |
+
+## Running the Example
+
+The `"zarrista"` engine is a Rust extension, so it is deliberately left out of
+this example's PEP 723 inline-dependency header — resolving it would put a
+toolchain build in the path of every run. The zarrista section is guarded and
+skips when the package is absent.
+
+Run it from a checkout of this branch:
+
+```bash
+uv run python examples/open_with_engine/open_with_engine.py
+```
+
+To exercise the Rust-backed engine, sync the `zarrista` dependency group:
+
+```bash
+uv run --group zarrista python examples/open_with_engine/open_with_engine.py
+```
+
+If `zarrista` is not installed, the example still runs and demonstrates the
+default engine; the `zarrista` portion is skipped.

examples/open_with_engine/open_with_engine.py

@@ -0,0 +1,120 @@
+# /// script
+# requires-python = ">=3.12"
+# dependencies = [
+#   "zarr @ git+https://github.com/zarr-developers/zarr-python.git@main",
+# ]
+# ///
+#
+# `zarrista` is deliberately *not* pinned here: it is a Rust extension, so
+# resolving it would put a toolchain build in the path of every run of this
+# example. The zarrista section below is guarded and skips when the package is
+# absent, so the example still runs and demonstrates the default engine. To see
+# the Rust-backed engine, run it from a checkout with that dependency group:
+#
+#     uv run --group zarrista python examples/open_with_engine/open_with_engine.py
+
+"""
+Open the same array data with a different backend ("engine").
+
+zarr-python routes an array's data I/O through a selectable *engine*. The engine
+is purely an *execution* setting: it chooses which compute backend reads and
+writes the array's chunks. The bytes on disk are identical regardless of the
+engine -- the same Zarr array, just a different machine doing the work.
+
+Engines demonstrated here:
+
+- `"default"` -- the built-in engine (used when `engine=` is omitted). Works on
+  every store and format.
+- `"zarrista"` -- a Rust-backed engine (via the `zarrista` package) that serves
+  Zarr v3 arrays on a `LocalStore` or an obstore-backed `ObjectStore`. Guarded:
+  skipped if the package is not installed.
+
+We write one array once, then open and read it back through each engine and
+assert the results are byte-for-byte identical.
+"""
+
+import tempfile
+from pathlib import Path
+
+import numpy as np
+
+import zarr
+from zarr.storage import LocalStore
+
+
+def zarrista_available() -> bool:
+    """Report whether the optional Rust-backed `"zarrista"` engine can be used.
+
+    The `"zarrista"` engine imports the `zarrista` package lazily, so we probe
+    that package directly: importing `zarr.zarrista` alone succeeds even when the
+    package is absent (the actual `ImportError` would only surface on first I/O).
+    """
+    try:
+        import zarrista  # noqa: F401
+    except ImportError:
+        return False
+    return True
+
+
+def main() -> None:
+    # Discover which engines exist. `zarr.list_engines()` returns the built-in
+    # engine names; `"zarrista"` additionally requires the `zarrista` package.
+    print("available engines:", zarr.list_engines())
+
+    # zarrista ingests a LocalStore (used here, on a temp directory) or an
+    # obstore-backed ObjectStore. We keep a single store so every engine reads
+    # the exact same bytes off the same disk.
+    with tempfile.TemporaryDirectory() as tmp:
+        store = LocalStore(Path(tmp) / "store")
+
+        # The data we will write once and read back through several engines.
+        data = np.arange(8 * 8, dtype="uint16").reshape(8, 8)
+
+        # --- Write the array once, with the default engine. -----------------
+        # No engine= here, so this uses the "default" engine.
+        source = zarr.create_array(
+            store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16"
+        )
+        source[:] = data
+
+        # --- 1. Default engine (the baseline). ------------------------------
+        # Either omit engine= or pass engine="default"; both mean the built-in.
+        default = zarr.open_array(store=store, path="a", engine="default")
+        # `array.engine` is the resolved engine instance backing this array.
+        print("default  engine:", type(default.engine).__name__)
+        np.testing.assert_array_equal(default[:], data)
+        # Basic indexing (ints and slices) is what engines route; check a slice.
+        np.testing.assert_array_equal(default[2:6, 1:5], data[2:6, 1:5])
+        print("default  (engine='default')   : read back OK")
+
+        # --- 2. zarrista engine (Rust), guarded behind availability. --------
+        if zarrista_available():
+            zst = zarr.open_array(store=store, path="a", engine="zarrista")
+            print("zarrista engine:", type(zst.engine).__name__)
+            np.testing.assert_array_equal(zst[:], data)
+            np.testing.assert_array_equal(zst[2:6, 1:5], data[2:6, 1:5])
+            # Same bytes on disk, different compute backend: identical to default.
+            np.testing.assert_array_equal(zst[:], default[:])
+            print("zarrista (engine='zarrista')  : read back OK, equals default")
+
+            # Writes also route through the engine. Create + write + read back
+            # entirely through zarrista, then confirm a *default* reader agrees.
+            written = zarr.create_array(
+                store=store,
+                name="b",
+                shape=(8, 8),
+                chunks=(4, 4),
+                dtype="uint16",
+                engine="zarrista",
+            )
+            written[:] = data
+            np.testing.assert_array_equal(zarr.open_array(store=store, path="b")[:], data)
+            print("zarrista (write path)         : round-trip OK, default reader agrees")
+        else:
+            print("zarrista                      : SKIPPED (package not installed)")
+
+    print("\nAll engines returned identical data. Same bytes on disk, different backend.")
+
+
+if __name__ == "__main__":
+    main()

mkdocs.yml

@@ -30,6 +30,7 @@ nav:
       - user-guide/glossary.md
   - Examples:
       - user-guide/examples/custom_dtype.md
+      - user-guide/examples/open_with_engine.md
       - user-guide/examples/rectilinear_chunks.md
       - user-guide/examples/codec_pipeline_performance.md
       - user-guide/examples/sharding_coalescing.md
@@ -40,6 +41,7 @@ nav:
         - api/zarr/abc/index.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.buffer</code>': api/zarr/abc/buffer.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.codec</code>': api/zarr/abc/codec.md
+        - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.engine</code>': api/zarr/abc/engine.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.metadata</code>': api/zarr/abc/metadata.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.numcodec</code>': api/zarr/abc/numcodec.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.store</code>': api/zarr/abc/store.md
@@ -93,6 +95,7 @@ nav:
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.testing.store</code>': api/zarr/testing/store.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.testing.strategies</code>': api/zarr/testing/strategies.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.testing.utils</code>': api/zarr/testing/utils.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.zarrista</code>': api/zarr/zarrista.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.zeros</code>': api/zarr/functions/zeros.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.zeros_like</code>': api/zarr/functions/zeros_like.md
       # The companion packages are Read the Docs subprojects of this one; link

pyproject.toml

@@ -145,6 +145,20 @@ dev = [
     "universal-pathlib",
     "mypy==2.3.0",
 ]
+zarrista = [
+    {include-group = "test"},
+    # Tracked from git rather than PyPI: the engine uses `store_array_subset`
+    # and the `Tensor` family, and `uv.lock` pins the exact commit.
+    "zarrista @ git+https://github.com/developmentseed/zarrista",
+    # The zarrista engine resolves selections through zarr-indexing's
+    # `LazyArray`; zarr itself does not depend on it.
+    "zarr-indexing>=0.2.1",
+    # zarrista's `store` submodule unconditionally imports both of these
+    # (for its `AsyncStore` type alias), even though neither is declared in
+    # zarrista's own package metadata as a runtime dependency.
+    "icechunk>=1.1.21",
+    "obstore>=0.10.1",
+]
 
 [tool.coverage.report]
 exclude_also = [

src/zarr/__init__.py

@@ -36,6 +36,7 @@
 )
 from zarr.core.array import Array, AsyncArray
 from zarr.core.config import config
+from zarr.core.engine import list_engines
 from zarr.core.group import AsyncGroup, Group
 
 # in case setuptools scm screw up and find version to be 0.0.0
@@ -164,6 +165,7 @@ def set_format(log_format: str) -> None:
     "full",
     "full_like",
     "group",
+    "list_engines",
     "load",
     "ones",
     "ones_like",