Skip to content

fix: accept universal-pathlib UPath as a StoreLike value - #280

Open
d-v-b wants to merge 15 commits into
mainfrom
claude/zarr-python-issue-4244-3863cc
Open

fix: accept universal-pathlib UPath as a StoreLike value#280
d-v-b wants to merge 15 commits into
mainfrom
claude/zarr-python-issue-4244-3863cc

Conversation

@d-v-b

@d-v-b d-v-b commented Aug 10, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Closes zarr-developers#4244.

What was going on

Passing a remote UPath to the top-level API raised TypeError: Unsupported type for store_like: 'S3Path'. Nothing changed on zarr's side — universal-pathlib 0.3.0 (2025-09-29) rebased UPath onto pathlib_abc, and remote paths like S3Path stopped being pathlib.Path subclasses and stopped implementing __fspath__.

That broke two things that had been working by accident:

  1. Via xarray (how the issue was reported). xarray converts any os.PathLike store to a string before handing it to zarr, so zarr used to receive "s3://..." and route it to FsspecStore.from_url. Zarr never actually saw a UPath. With no __fspath__, the raw object now comes through.
  2. Direct zarr.open(S3Path). I confirmed against zarr 3.1.3 + upath 0.2.6 that this used to hit the isinstance(store_like, Path) branch and build a LocalStore rooted at an S3Path — which worked only because upath 0.2 overrode every pathlib method to dispatch through fsspec.

So the report is filed as a feature request, and that framing is accurate: remote UPath support was never deliberate. But FsspecStore.from_upath already exists and universal-pathlib is already an optional dependency, so wiring it up is small.

The change

  • make_store gains a UPath branch. Remote UPaths go to FsspecStore.from_upath; local ones (protocol "" or "file") go to a LocalStore, so UPath('/data') and Path('/data') agree, mirroring how the equivalent strings are already routed.

  • The branch is checked before the Path branch. On universal-pathlib 0.2 every UPath — including S3Path — subclasses pathlib.Path, so otherwise remote paths would keep landing in a LocalStore there. I verified routing is now identical on 0.2.0 and 0.3.10:

    value 0.2.0 0.3.10
    UPath('s3://bucket/foo.zarr') FsspecStore FsspecStore
    UPath('memory://bucket/foo.zarr') FsspecStore FsspecStore
    UPath('file:///tmp/x.zarr') LocalStore LocalStore
    UPath('/tmp/x.zarr') LocalStore LocalStore
  • Drive-by fix: FsspecStore.from_upath passed upath.fs straight to the constructor without going through _make_async (unlike from_mapper). A UPath hands back a filesystem in whatever mode it was built with, which is synchronous unless the caller passed asynchronous=True. So from_upath raised TypeError: Filesystem needs to support async operations. for genuinely sync filesystems, and emitted the "not created with asynchronous=True" warning for sync-mode instances of async ones. Both reproduce on upath 0.3.10.

  • storage_options passed alongside a UPath still raises, since the UPath carries its own — the existing check already covers this, and there's now a test pinning it.

Notes for review

  • Passing a UPath with storage_options is an error rather than a merge. Ambiguous either way; erroring seemed better than silently picking one. Happy to change. Note xarray always forwards storage_options to zarr, but as None when unset, so the reported case is unaffected.
  • Local UPaths are routed to LocalStore rather than through fsspec's LocalFileSystem. This is a judgement call for consistency with the string handling (and it's faster); the alternative is to send everything to FsspecStore and keep the branch simpler.
  • The UPath re-export in _utils.py needed an explicit as UPath alias to satisfy no_implicit_reexport.

Testing

  • New parametrized routing test over the four cases in the table, plus an error-case test for storage_options.
  • New moto-backed S3 tests: an end-to-end open_group(UPath(...)) round trip reproducing the issue, and one asserting from_upath on a sync-mode UPath yields an async filesystem with storage options preserved and no warning.
  • I checked the new tests fail without the source change (5 of 6 do; the other two cases are regression guards that already passed).
  • One new test needed a filterwarnings for a DeprecationWarning that universal-pathlib 0.2.0 raises from its own subclass registry when any local UPath is constructed. It fires on a bare UPath('/tmp/x') with zarr uninvolved, and would otherwise break the min_deps CI env, which pins universal_pathlib==0.2.0.
  • Full suite: 7069 passed, 380 skipped, 4 xfailed. ruff and mypy clean. Store tests also verified against universal-pathlib 0.2.0.

🤖 Generated with Claude Code

dependabot Bot and others added 15 commits May 31, 2026 19:28
…#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
A remote UPath now creates an FsspecStore from the filesystem and storage
options the UPath already carries, and a local UPath creates a LocalStore so
that UPath("/data") and Path("/data") agree.

This previously worked only by accident. In universal-pathlib < 0.3 every
UPath subclassed pathlib.Path and implemented __fspath__, so a remote path was
either converted to a URI string by the caller (xarray does this) or wrapped
in a LocalStore that happened to dispatch through fsspec. Since
universal-pathlib 0.3 remote paths do neither, and passing one raised
TypeError: Unsupported type for store_like.

The UPath branch is checked before the Path branch so that routing is
identical on both universal-pathlib 0.2 and 0.3.

FsspecStore.from_upath now converts the UPath's filesystem to async mode
instead of raising TypeError for synchronous filesystems and warning for
sync-mode instances of async ones.

The memory:// routing test wraps a synchronous MemoryFileSystem, which needs
fsspec's AsyncFileSystemWrapper, so it is skipped below fsspec 2024.12.0 as
test_wrap_sync_filesystem already does.

Closes zarr-developers#4244

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
d-v-b force-pushed the claude/zarr-python-issue-4244-3863cc branch from 63e3930 to e023851 Compare August 10, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Universal-pathlib support for zarr.open

1 participant